Statistical Inference

Ery Arias-Castro
University of California, San Diego
Hypothesis testing
Consider the problem of deciding
whether a roulette is fair to the
player or favors the casino.
We could physically examine the roulette.
I Doing this meaningfully is likely very involved and requires
particular expertise and equipment.
I Doing this might not be relevant as it might ignore how the
roulette is used in practice.

This is emblematic of what happens in practice in important

areas of applications such as clinical trials.

In clinical trials, the second issue is closely related to the intention-to-treat

approach to analysis, which focuses on how a treatment is deployed in practice
as opposed to how it is deployed in an idealized setting.
An American roulette has 38
slots: 18 are black, 18 are red,
and 2 are green. A ball is rolled,
and eventually lands in one of
the slots.

Players can make a number of bets, for example, on color (red or

black) or one particular number (‘straight up’). Although it is
possible for a player to place a bet that includes 0 and/or 00, we
assume for simplicity that this is not the case.
In that case, the wheel is fair (to the players) when the probability
of a given spin resulting in green is
2
p0 =
38
From the perspective of players, what matters is the proportion of
times that the ball lands in a green slot during normal operation.

Therefore, one way to assess whether a roulette is fair or not is to

observe a number of spins during normal operation and record the
number of times that the ball lands in a green slot. The number of
spins is set beforehand to some value denoted n.
Probability experiment

Denote the result of the spins by

x1 , . . . , x n

where 
1 if the ith spin results in green
xi =
0 otherwise

If the roulette is fair, x1 , . . . , xn is the realization of

X1 , . . . , Xn , independent Bernoulli(p0 ) random variables

If we do not question the independence of the spins, regardless of
whether the roulette is fair or not,

X1 , . . . , Xn are independent Bernoulli(p? ) random variables

where
p? = probability of a spin resulting in green

Adopting this perspective, we have a well-defined probability model

where the roulette being fair translates into

p? = p0

The question we have is whether this is the case or not.

Hypothesis testing problem

Typically, the default hypothesis is that the roulette is fair, which

leads formalizing the question as the hypothesis testing problem:

H0 : p? = p0 versus H1 : p? > p0
| {z } | {z }
null hypothesis alternative hypothesis

The alternative above is said to be one-sided and models situations

favoring the casino.
Test function

The analyst needs to decide, based on an inspection of the data, if

there is enough evidence against the null hypothesis.

Such a decision is formalized as a test, which is formally a function

of the data with values in {0, 1}:

I The test returns 0 if it deems there is not enough evidence

against the null hypothesis.

I The test returns 1 if it deems there is enough evidence against

the null hypothesis (in favor of the alternative). We then say the
test rejects the null hypothesis.
A test can make one of two types of error:

Null is true Null is false

Rejection (test = 1) type I error correct

No rejection (test = 0) correct type II error

In other words:

type I error = incorrect rejection = false positive

type II error = incorrect failure to reject = false negative
 It is trivial to control the probability of either type of error.

• The test that never rejects (test constant at 0) never makes a

type I error... but it is not useful!
• The test that always rejects (test constant at 1) never makes
a type II error... but it is not useful!

It is not possible to control both the probability of type I error

and the probability of type II error.

Traditionally, the probability of type I error is controlled at

some prescribed level α.

Setting α = 0.05 = 5% is common, but the most appropriate choice depends

on the application. Almost always, α ≤ 0.10 = 10%.
Testing via an estimator

In the present example, the data consist of x1 , . . . , xn , assumed to

be the realization of X1 , . . . , Xn , independent Bernoulli(p? )
random variables, where

p? = probability of a spin resulting in green

We are tackling the following hypothesis testing problem:

2
H0 : p? = p0 = 38 versus H1 : p? > p0

A natural estimate for p? is its empirical equivalent

p̂ = proportion of spins that resulted in green in the sample

This is the plug-in principle.

Due to chance variation, in a particular run of the experiment, it is
(very) unlikely that p̂ = p? . In particular, it is quite unlikely that
p̂ = p0 even when the null hypothesis is true.

However, it is the case that p̂ will ‘tend to be closer’ to p0 under

the null hypothesis than under the alternative.

The idea is to study the deviations of p̂ from p0 under the

null hypothesis. If in practice the observed deviation is larger
than would typically be the case under the null hypothesis,
we have evidence against the null hypothesis.
We therefore want to build

a test that rejects for large values of p̂

Thus p̂ plays the role of test statistic and ‘large’ is understood

relative to typical values of p̂ under the null hypothesis.

Equivalently, we are considering rejecting for large values of

Y = np̂ = number of greens in the sample

= X1 + · · · + Xn
In general, Y has the binomial(n, p? ) distribution.
I Under H0 , Y has the binomial(n, p0 ) distribution.

We know the distribution of Y under the null hypothesis.

0.6
0.5
0.4
0.3
0.2
0.1
0.0

0 1 2 3 4 5 6 7 8 9 10

Null density of Y when n = 10

0.3
0.2
0.1
0.0

0 5 10 15 20

Null density of Y when n = 20

0.25
0.20
0.15
0.10
0.05
0.00

0 10 20 30 40 50

Null density of Y when n = 50

0.15
0.10
0.05
0.00

0 20 40 60 80 100

Null density of Y when n = 100

0.12
0.10
0.08
0.06
0.04
0.02
0.00

0 50 100 150 200

Null density of Y when n = 200

We are considering a test of the form

1 if Y > c
=
0 otherwise

c is called the critical value.

To control the level at α, the critical value c is chosen smallest

such that under the null hypothesis the probability of rejecting
is upper bounded by α. That way, the test rejects as much
as possible while satisfying the appropriate level.
Therefore, c is smallest such that


P0 Y > c ≤ α

where P0 denotes the probability under H0 .

Equivalently, c is smallest such that


P0 Y ≤ c ≥ 1 − α

so that c is a type of (1 − α)-quantile of the distribution of Y

under the null hypothesis.

α n= 10 20 50 100 200
0.10 1 2 5 8 15
0.05 2 3 5 9 16
0.01 3 4 7 11 18
0.6
0.5
0.4
0.3
0.2
0.1
0.0

0 1 2 3 4 5 6 7 8 9 10

α = 0.05 and n = 10
0.3
0.2
0.1
0.0

0 5 10 15 20

α = 0.05 and n = 20
0.25
0.20
0.15
0.10
0.05
0.00

0 10 20 30 40 50

α = 0.05 and n = 50
0.15
0.10
0.05
0.00

0 20 40 60 80 100

α = 0.05 and n = 100

0.12
0.10
0.08
0.06
0.04
0.02
0.00

0 50 100 150 200

α = 0.05 and n = 200

P-value

We may not want to test at a particular level, but may prefer to

quantify the amount of evidence against the null hypothesis.

The p-value is the largest level α at which the test fails to reject,
or equivalently, smallest level α at which the test rejects.
 For a test procedure that rejects for large values of a statistic,
the p-value is the probability under the null hypothesis of
observing a value of the test statistic at least as large as the
observed value.

Returning to our example, consider rejecting for large values of Y .

If the observed value of Y is y,

p-value = P0 (Y ≥ y)


= Prob binomial(n, p0 ) ≥ y
For a numerical example, if we spin the roulette n = 50 times and
observe the ball landing green y = 5 times,

2


p-value = Prob binomial(50, 38 ) ≥ 5 ≈ 0.12 = 12%

This means that, if the null hypothesis were to be true, there

would be 12% chance of observing such a large number of greens.
This provides some evidence against the null hypothesis, although most would
consider this to be weak evidence.
 Continuing with the game of
roulette, suppose a player wants
to know if for a particular
roulette betting on red offers the
same odds as betting on black.

This is the case if the roulette is fair, by which we here mean that
the ball falls in any slot with the same probability, as there are the
same number of red and black slots (18 each).
As before, we could physically examine the roulette, but again the
same issues arise. But it is simpler and likely more relevant and
accurate to observe a number of spins during normal operation.

Since a spin where the ball lands on a green slot does not bring
additional information pertinent to the question, we simply ignore
such spins. The design is thus a little different: We spin the
roulette until the number of spins that resulted in either red or
black reaches a predetermined value n.
Probability experiment

Denote the result of the relevant spins by

x1 , . . . , x n

where 
1 if the ith spin results in red
xi =
0 if the ith spin results in black
(Again, a relevant spin is either red or black.)

If the roulette is fair, x1 , . . . , xn is the realization of

X1 , . . . , Xn , independent Bernoulli(1/2) random variables

In what follows, we let p0 = 1/2.

If we do not question the independence of the spins, regardless of
whether the roulette is fair or not,

X1 , . . . , Xn are independent Bernoulli(p? ) random variables

where
p? = probability of a spin resulting in red

Adopting this perspective, we have a well-defined probability model

where the roulette being fair translates into p? = p0 . The question
we have is whether this is the case or not.
Hypothesis testing problem

Adopting the default position that the roulette is fair, we formalize

the question as the following hypothesis testing problem:

H0 : p? = p0 versus H1 : p? 6= p0

The alternative above is said to be two-sided.

Testing via an estimator

The plug-in principle leads us to estimate

p? = probability of a spin resulting in red

with its empirical counterpart

p̂ = proportion of spins that resulted in red in the sample

The idea is to study the deviations of p̂ from p0 under the null

hypothesis. If the observed deviation is larger than is typical
under the null hypothesis, we have some evidence against the
null hypothesis.
One way of formalizing this idea is to build

a test that rejects for large values of |p̂ − p0 |

Thus |p̂ − p0 | plays the role of test statistic and ‘large’ is

understood relative to typical values under the null hypothesis.

Equivalently, we are rejecting for large values of |Y − np0 |, where

Y = np̂ = number of reds in the sample

= X1 + · · · + Xn
In general, Y has the binomial(n, p? ) distribution.
I Under H0 , Y has the binomial(n, p0 ) distribution.

We know the distribution of Y under the null hypothesis.

Knowing the null distribution of Y , we also know the null

distribution of |Y − np0 |.

In the present case, we can exploit the fact that the null distribution of Y ,
which is binomial(n, 1/2), is symmetric about n/2, to get to the null
distribution of |Y − n/2| analytically. Indeed, it is the case that for c > 0,

P0 |Y − n/2| > c = 2 × Prob binomial(n, 21 ) > n/2 + c

But this is not particularly important in practice, as there are alternative

numerical means.
0.25
0.20
0.15
0.10
0.05
0.00

0 1 2 3 4 5 6 7 8 9 10

Null density of Y when n = 10

0.15
0.10
0.05
0.00

0 5 10 15 20

Null density of Y when n = 20

0.10
0.08
0.06
0.04
0.02
0.00

0 10 20 30 40 50

Null density of Y when n = 50

0.08
0.06
0.04
0.02
0.00

0 20 40 60 80 100

Null density of Y when n = 100

0.05
0.04
0.03
0.02
0.01
0.00

0 50 100 150 200

Null density of Y when n = 200

We are considering a test of the form

1 if |Y − np | > c
0
=
0 otherwise

where c is smallest such that

P0 |Y − np0 | > c ≤ α

or, equivalently, c is smallest such that

P0 |Y − np0 | ≤ c ≥ 1 − α

α n= 10 20 50 100 200
0.10 3 4 6 8 12
0.05 3 4 7 10 14
0.01 4 6 9 13 18
0.25
0.20
0.15
0.10
0.05
0.00

0 1 2 3 4 5 6 7 8 9 10

α = 0.05 and n = 10
0.15
0.10
0.05
0.00

0 5 10 15 20

α = 0.05 and n = 20
0.10
0.08
0.06
0.04
0.02
0.00

0 10 20 30 40 50

α = 0.05 and n = 50
0.08
0.06
0.04
0.02
0.00

0 20 40 60 80 100

α = 0.05 and n = 100

0.05
0.04
0.03
0.02
0.01
0.00

0 50 100 150 200

α = 0.05 and n = 200

P-value

Recall that, for a test procedure that rejects for large values of a
statistic, the p-value is the probability under the null hypothesis of
observing a value of the test statistic at least as large as the
observed value.

In our example, if the observed value of Y is y,

p-value = P0 |Y − np0 | ≥ |y − np0 |
= Prob |binomial(n, 21 ) − n( 12 )| ≥ |y − n( 12 )|


For a numerical example, if out of a total of n = 50 spins resulting
in either red or black, we observe y = 17 reds,

p-value = Prob |binomial(50, 21 ) − 50( 12 )| ≥ |17 − 50( 21 )|

= Prob |binomial(50, 21 ) − 25| ≥ 8 ≈ 0.033

This means that, if the null hypothesis were to be true, there

would be 3.3% chance of observing Y so far away from its mean.
This provides some evidence against the null hypothesis. In some cases, this
might be considered fairly strong evidence.
Normal approximation

Although using software we can easily work directly with the

binomial family of distributions, we may use a normal
approximation to simplify things a little bit.

The simplification that ensue was more useful in the pre-

computer age, but it is still common in introductory Statistics
courses to use a normal approximation.
We illustrate the use of a normal approximation in the last setting.

To choose a critical value or compute a p-value, the reference

distribution is binomial(n, p0 ) with p0 = 1/2.

The normal approximation to the binomial replaces the

binomial(n, p0 ) distribution with the normal distribution with same
p
mean np0 and same standard deviation np0 (1 − p0 ).
0.25
0.20
0.15
0.10
0.05
0.00

0 2 4 6 8 10

Normal approximation to binomial(n, 21 ) when n = 10

0.15
0.10
0.05
0.00

0 5 10 15 20

Normal approximation to binomial(n, 21 ) when n = 20

0.10
0.08
0.06
0.04
0.02
0.00

0 10 20 30 40 50

Normal approximation to binomial(n, 21 ) when n = 50

0.08
0.06
0.04
0.02
0.00

0 20 40 60 80 100

Normal approximation to binomial(n, 12 ) when n = 100

0.05
0.04
0.03
0.02
0.01
0.00

0 50 100 150 200

Normal approximation to binomial(n, 12 ) when n = 200

The normal approximation gives that, under the null hypothesis,
Y − np0
p is approximately standard normal
np0 (1 − p0 )

We can use this to approximate the p-value as follows: having

observed Y to be y, we let
y − np0
z=p ← called z-score
np0 (1 − p0 )

and derive


p-value = P0 |Y − np0 | ≥ |y − np0 |
!
|Y − np0 | |y − np0 |
= P0 p ≥p
np0 (1 − p0 ) np0 (1 − p0 )


≈ Prob |standard normal| ≥ |z|


= 2 × Prob standard normal ≥ |z| ← by symmetry
Consider the numerical example where out of a total of n = 50
spins resulting in either red or black, we observe y = 17 reds.
We derived the exact p-value earlier

Prob |binomial(50, 12 ) − 50 · 12 | ≥ |17 − 50 · 12 | ≈ 0.033

With a normal approximation, we first compute the z-score

17 − 50 · 12
z=q ≈ −2.26
50 · 12 (1 − 12 )

and then derive

2 × Prob standard normal ≥ 2.26 ≈ 0.024
Confidence intervals
Point estimator

In the last setting, suppose we want to estimate

p? = probability of a spin resulting in red

The plug-in principle leads us to use its empirical counterpart

p̂ = proportion of spins that resulted in red in the sample

In this context, p̂ is called a point estimator or simply estimator,

that is, a statistic meant to estimate a particular parameter.
For a numerical example, if out of a total of n = 50 spins resulting
in either red or black, we observe y = 17 reds, then our estimate
for the parameter p is
Y 17
p̂ = = = 0.34 = 34%
n 50
While an ‘estimator’ is a statistic, ‘estimate’ refers to the value of an estimator
when evaluated on a particular sample or data set.

Beyond a single (or ‘point’) value, we would like to have a

range of values where we are ‘likely’ to find the true value of
the parameter.
Confidence interval

A confidence interval is an interval that is a function of the data,

therefore, of the form

I = [L, U ] where L and U are statistics

A confidence interval is said to have level 1 − α for a parameter if

it contains the parameter with probability at least 1 − α.

In our example, I is a confidence interval at level 1 − α for p∗ if

P(p∗ ∈ I) ≥ 1 − α

If I = [L, U ], then this is equivalent to

P(L ≤ p∗ ≤ U ) ≥ 1 − α
Obtaining a confidence interval from a test

One way to obtain a confidence interval is via a testing procedure:

At level 1 − α, the interval consists of all values of the pa-

rameter p0 where the test fails to reject the null hypothesis
p∗ = p0 at level α.

As defined, the ‘interval’ may in fact be the union of multiple intervals in

general, although it is a single interval in a number of important situations.

In our example,

I = p0 : test does not reject p∗ = p0 at level α

The computation of this interval is typically done with software.

Obtaining a confidence interval from an estimator

We can also obtain a confidence interval from an estimator.

The idea is to study the deviations of the estimator and un-

derstand how far it tends to be from the parameter.
In our context, we consider the estimator p̂ and study its deviations
from p∗ . For simplicity, we do so using a normal approximation.

We know that Y = np̂ is binomial(n, p∗ ), so that

Y − np∗
p is approximately standard normal
np∗ (1 − p∗ )

or, equivalently, that

p̂ − p∗
p is approximately standard normal
p∗ (1 − p∗ )/n

or, equivalently, that p̂ is approximately normal with mean p∗ and

p
standard deviation p∗ (1 − p∗ )/n.
Let zα be the α-quantile of the standard normal distribution
defined by


Prob standard normal ≤ zα = α

α 0.10 0.05 0.025 0.01 0.005

zα -1.28 -1.64 -1.96 -2.33 -2.58

It can be seen that

Prob zα/2 ≤ standard normal ≤ z1−α/2 = 1 − α

and because the standard normal distribution is symmetric, we

have zα/2 = −z1−α/2 , and this is equivalent to


Prob |standard normal| ≤ z1−α/2 = 1 − α
We thus have that
!
p̂ − p∗
P p ≤ z1−α/2 ≈1−α
p∗ (1 − p∗ )/n

or, in other words,

p̂ − p∗
p ≤ z1−α/2
p∗ (1 − p∗ )/n

with probability ≈ 1 − α.

The accuracy depends on the sample size n and the value of p∗ . The latter is
unknown. A common rule of thumb for assessing whether the interval is
‘accurate enough’ is the requirement

np̂ ≥ 10 and n(1 − p̂) ≥ 10

Some manipulations give that
p̂ − p∗
p ≤ z1−α/2
p∗ (1 − p∗ )/n
is equivalent to
L ≤ p∗ ≤ U
for some complicated but known bounds L and U .

Therefore, if I = [L, U ], we have that

P (p∗ ∈ I) ≈ 1 − α
so that I is a confidence interval for p∗ at approximate level 1 − α.

In more detail,
2 2
p̂ + z1−α/2 /2n z1−α/2 p̂(1 − p̂) + z1−α/2 /4
L= 2
− √ 2
1 + z1−α/2 /n n (1 + z1−α/2 /n)2
2 2
p̂ + z1−α/2 /2n z1−α/2 p̂(1 − p̂) + z1−α/2 /4
U = 2
+ √ 2
1+ z1−α/2 /n n (1 + z1−α/2 /n)2
A further simplification is possible based on the fact that
p̂ − p∗
p is approximately standard normal
p̂(1 − p̂)/n

so that
p̂ − p∗
p ≤ z1−α/2
p̂(1 − p̂)/n
with probability ≈ 1 − α.
Straightforward manipulations yield that this is equivalent to
r r
p̂(1 − p̂) p̂(1 − p̂)
p̂ − z1−α/2 ≤ p∗ ≤ p̂ + z1−α/2
n n
with probability ≈ 1 − α, so that
r
p̂(1 − p̂)
p̂ ± z1−α/2
n
is a confidence interval for p∗ with approximate level 1 − α.

There is no advantage to using this simpler interval unless

computations need to be done by hand.
Earlier, we could have simply stated that p∗ is in the interval
r
p∗ (1 − p∗ )
p̂ ± z1−α/2
n
with probability ≈ 1 − α.

But this is not a confidence interval as it depends on p∗ and therefore cannot

be computed from the dataset alone.
For a numerical example, suppose that out of a total of n = 50
spins resulting in either red or black, we observe y = 17 reds.
Using these data, we want to build a 90% confidence interval for
p∗ . Note that α = 0.10.

Using the test directly based on the deviations of the binomial

distribution, we can derive the following interval

[0.229, 0.466]
(This is not exactly the interval that we would get, but should be pretty close.)

Using the normal approximation without further simplification

[0.240, 0.456]

Using the normal approximation with simplification

[0.229, 0.451]
Obtaining a test from a confidence interval

We saw that we can obtain a confidence interval from a test.

We can also obtain a test from a confidence interval.

The test rejects if the hypothesized value of the parameter is

not in the confidence interval. If the interval has confidence
level 1 − α, then the test has level α.
In our example, for some given p0 consider the following hypothesis
testing problem:

H0 : p? = p0 versus H1 : p? 6= p0

With I being a confidence interval for p∗ , define the test that

rejects the null hypothesis if p0 ∈

/I

If I has level 1 − α, then this test has level α.