Introduction to Statistical Inference

Edwin Leuven
Introduction

Today:

I Define key terms that are associated with inferential statistics

I Revise concepts related to random variables, the sampling
distribution and the Central Limit Theorem

2/45
Introduction

Until now we’ve mostly dealt with descriptive statistics and with
probability
In descriptive statistics one investigates the characteristics of the
data

I using graphical tools and numerical summaries

The frame of reference is the observed data

3/45
Introduction

4/45
Introduction

5/45
Introduction

6/45
Introduction

7/45
Introduction

In probability, the frame of reference is all data sets that could have
potentially emerged from a population
The aim of statistical inference is to learn about the population
using the observed data
This involves:

I computing something with the data

I a statistic: function of data
I interpret the result
I in probabilistic terms: sampling distribution of statistic

8/45
Introduction

Probability

Population Sample (Data)

fX (x ) (x1 , . . . , xn )

Parameter Statistic
x̄ = n1 ni=1 xi
P
E [X ] = µ

Inference

9/45
Point estimation

We want to estimate a population parameter using the observed

data

I f.e. some measure of variation, an average, min, max, quantile

Point estimation attempts at a best guess for the value of that

parameter
An estimator is a guess based on a function of the data (statistic)
We usually mean by “best” an estimator that is likely to be closer to
the population parameter value than other estimators
Hence, the choice of a specific statistic as an estimator depends on
the probabilistic characteristics of this statistic in the context of the
sampling distribution

10/45
Confidence Interval

We can also quantify the uncertainty (sampling distribution) of our

point estimate
One way of doing this is by constructing an interval that is likely to
contain the population parameter.
One such an interval, which is computed on the basis of the data, is
called a confidence interval
The sampling probability that the confidence interval will indeed
contain the parameter value is called the confidence level
We construct confidence intervals for a given confidence level

11/45
Hypothesis Testing

The scientific paradigm involves the proposal of new theories that

presumably provide a better description of the laws of Nature
If the empirical evidence is inconsistent with the predictions of the
old theory but not with those of the new theory

I then the old theory is rejected in favor of the new one

I otherwise, the old theory maintains its status

Statistical hypothesis testing is a formal method for determining

which of the two hypothesis should prevail that uses this paradigm

12/45
Statistical hypothesis testing

Each of the two hypothesis, the old and the new, predicts a different
distribution for the empirical measurements
In order to decide which of the distributions is more in tune with the
data a statistic is computed
This statistic t is called the test statistic
A threshold c is set and the old theory is reject if t > c
Hypothesis testing consists in asking a binary question about the
sampling distribution of t

13/45
Statistical hypothesis testing

This decision rule is not error proof, since the test statistic may fall
by chance on the wrong side of the threshold
Suppose we know the sampling distribution of the test statistic t
We can then set the probability of making an error to a given level
by setting c
The probability of erroneously rejecting the currently accepted
theory (the old one) is called the significance level of the test
The threshold is selected in order to assure a small enough
significance level

14/45
Multiple measurements

The method of testing hypothesis is also applied in other practical

settings where it is required to make decisions
Consider a random trial of a new treatment to a medical condition
where the

I treated get the new treatment

I controls get the old treatment

and measure their response

We now have 2 measurements that we can compare.
We will use statistical inference to make a decision about whether
the new treatment is better

15/45
Statistics

Statistical inferences, be it point estimation, confidence intervals, or

hypothesis tests, are based on statistics computed from the data
A statistic is a formula which is applied to the data
We think of it as a statistical summary of the data
Examples of statistics are

I the sample average and

I the sample standard deviation

For a given dataset a statistic has a single numerical value

It will be different for a different random sample!
The statistic is therefore a random variable

16/45
Statistics

It is important to distinguish between

1. the statistic (a random variable)

2. the realisation of the statistic for a given sample (a number)

we therefore denote the statistic with capitals, f.e. the sample mean:
1 Pn
I X̄ = n i=1 Xi

and the realisation of the statistic with small letters:

1 Pn
I x̄ = n i=1 xi

17/45
Example: Polling

The elections in 2009 How will

the next elections look?

18/45
Example: Polling

Answer: We ask a bunch of people!

19/45
Example: Polling

Imagine we want to predict whether the left block or the right block
will get a majority in parliament
Key quantities:

I N = 4,166,612 - Population
I p = (# people who support the right) / N
I 1 − p = (# people who support the left) / N

We can ask the following questions:

1. What is p?
2. Is p > 0.5?
3. We estimate p but are we sure?

20/45
Example: Polling
We poll a random sample of n = 1,000 people from the population
without replacement:

I choose person 1 at random from N, choose person 2 at random

from N-1 remaining, etc.
or, choose a random set of n people from all Nn = n!(N−n)!
N!


I
possible sets

Let (
1 if person i support the right
Xi =
0 if person i support the left
and denote our data by x1 , . . . , xn
Then we can estimate p by

p̂ = (x1 + . . . + xn )/n

21/45
Example: Polling

To construct the poll we randomly sampled the population

With a random sample each of the n people is equally likely to be
the ith person, therefore

E [Xi ] = 1 · p + 0 · (1 − p) = p

and therefore
E [p̂] = E [(X1 + . . . + Xn )/n]
= (E [X1 ] + . . . + E [Xn ])/n = p

The “average” value of p̂ is p, and we say that p̂ is unbiased

Unbiasedness refers to the average error over repeated sampling,
and not the error for the observed data!

22/45
Example: Polling

Say 540 support the right, so p̂ = 0.54

Does this mean that in the population:

I p = 0.54?
I p > 0.5?

The data are a realization of a random sample and p̂ is therefore a

random variable!
For a given sample we will therefore have estimation error

estimation error = p̂ − p 6= 0

which comes from the difference between our sample and the
population

23/45
Example: Polling

When sampling with replacement the Xi are independent, and

p(1−p)
I Var [p̂] = n

When sampling without replacement the Xi are not independent

N1 − 1 N1
= Pr(Xi = 1|Xj = 1) 6= Pr (Xi = 1|Xj = 0) =
N −1 N −1
and we can show that
 
p(1−p) n−1
I Var [p̂] = n 1− N−1

For N = 4, 166, 612, n = 1, 000, and p = 0.54, the standard

deviation of p̂ ≈ 0.016.
But what is the distribution of p̂?
24/45
The Sampling Distribution

Statistics vary from sample to sample

The sampling distribution of a statistic

I is the nature of this variability

I can sometimes be determined and often approximated

The distribution of the values we get when computing a statistic in

(infinitely) many random samples is called the sample distribution of
that statistic

25/45
The Sampling Distribution

We can sample from

I population
I eligible voters in norway today
I model (theoretical population)
I Pr(vote right block) = p

The sampling distribution of a statistic depends on the population

distribution of values of the variables that are used to compute the
statistic.

26/45
Sampling Distribution of Statistics

Theoretical models describe the distribution of a measurement as a

function of one or more parameters.
For example,

I in n trials with succes probability p, the total number of

successes follows a Binomial distribution with parameters n and
p
I if an event happens at rate λ per unit time then the probability
that k events occur in a time interval with length t follows a
Poission distribution with parameters λt and k

27/45
Sampling Distribution of Statistics

More generally the sampling distribution of a statistic depends on

I the sample size

I the sampling distribution of the data used to construct the
statistic

can be complicated!
We can sometimes learn about the sampling distribution of a
statistic by

I Deriving the finite sample distribution

I Approximation with a Normal distribution in large samples
I Approximation through numerical simulation

28/45
Finite sample distributions
Sometimes we can derive the finite sample distribution of a statistic
Let the fraction of people voting right in the population be p
Because we know the distribution of the data (up to the unknown
parameter p) we can derive the sampling distribution
In a random sample of size n the probability of observing k people
voting on the right can be derived and follows a binomial distribution
!
n k
Pr(X = k) = p (1 − p)n−k
k

This depends on p which is unknown.

This approach is however often not feasible
The statistic may be complicated, depend on different variables, the
population distribution of these variables is unknown
29/45
Theoretical Distributions of Observations (Models)

Distribution Sample Space f (x )

n
k n−k
Binomial 1, . . . , n k p (1 − p)
Poisson 1, 2, . . . λk exp(−λ)/k!
Uniform [a, b] 1/(b − a)
Exponential [0, ∞) λ exp(−λx )
Normal (−∞, ∞) √ 1 exp(− 1 ( x −µ )2 )
2πσ 2 σ

Distribution E [X ] Var (X ) R
Binomial np np(1 − p) d,p,q,rbinom
Poisson λ λ d,p,q,rpoisson
1 1 2
Uniform 2 (a + b) 12 (b − a) d,p,q,runif
Exponential λ−1 λ −2 d,p,q,rexp
Normal µ σ2 d,p,q,rnorm

30/45
Example: Polling
hist(
replicate(
10000,mean(rbinom(1000, 1, .54)))
, main="", xlab="p_hat",prob=TRUE,breaks=50)
25
20
Density

15
10
5
0

0.48 0.50 0.52 0.54 0.56 0.58 0.60

31/45
The Normal Approximation

In general, the sampling distribution of a statistic is not the same as

the sampling distribution of the measurements from which it is
computed.
If the statistic is

1. (a function of) a sample average and

2. the sample is large

then we can often approximate the sampling distribution with a

Normal distribution

32/45
Example: Polling

In the graph p̂ looked like it had a Normal distribution with mean

0.54 and s.d. 0.16
If N  n then Xi are approximately independent, and if n is large
then

√
n(p̂ − p) ∼ N(0, p(1 − p))
or equivalently
p(1 − p)
p̂ ∼ N(p, )
n
by the Central Limit Theorem

33/45
Example: Polling

curve(dnorm(x, mean=.54, sd=0.016),

col="darkblue", lwd=2, add=TRUE, yaxt="n")
25
20
Density

15
10
5
0

0.48 0.50 0.52 0.54 0.56 0.58 0.60

p_hat
34/45
Approximation through numerical simulation

Computerized simulations can be carried out to approximate

sampling distributions.
With a model we can draw many random samples, compute the
statistic, and characterize it’s sampling distribution.
Assume price ∼ Expontential(λ)
Consider samples of size n = 201
E [price] = λ−1 and Var [price] = λ−2
and therefore
q
Var (price) = (1/λ2 )/201 ≈ 0.0705/λ

35/45
Approximation through numerical simulation

Remember that 95% of the probability density of a normal

distribution is within 1.96 s.d. of its mean.
The Normal approximation for the sampling distribution of the
average price suggests

√
1/λ ± 1.96 · 1/(λ n)

should contain 95% of the distribution.

36/45
Approximation through numerical simulation

We may use simulations in order to validate this approximation

Assume λ = 1/12, 000

[Link] = replicate(10^5,mean(rexp(201,1/12000)))
mean(abs([Link]-12000) <= 1.96*0.0705*12000)

## [1] 0.95173

Which shows that the Normal approximation is adequate in this

example
How about other values of n or λ?

37/45
Approximation through numerical simulation

Simulations may also be used in order to compute probabilities in

cases where the Normal approximation does not hold.
Consider the following statistic

(min(xi ) + max(xi ))/2

where Xi ∼ Uniform(3, 7) and n = 100

What interval contains 95% of the observations?

38/45
Approximation through numerical simulation
Let us carry out the simulation that produces an approximation of
the central region that contains 95% of the sampling distribution of
the mid-range statistic for the Uniform distribution:

[Link] <- rep(0,10^5)

for(i in 1:10^5) {
X <- runif(100,3,7)
[Link][i] <- (max(X)+min(X))/2
}
quantile([Link],c(0.025,0.975))

## 2.5% 97.5%
## 4.9409107 5.0591218

Observe that (approximately) 95% of the sampling distribution of

the statistic are in the range [4.941680, 5.059004].
39/45
Approximation through numerical simulation

Simulations can be used in order to compute any numerical

summary of the sampling distribution of a statistic
To obtain the expectation and the standard deviation of the
mid-range statistic of a sample of 100 observations from the
Uniform(3, 7) distribution:

mean([Link])

## [1] 4.9998949

sd([Link])

## [1] 0.027876151

40/45
Approximation through numerical simulation

Computerized simulations can be carried out to approximate

sampling distributions.

1. draw a random sample of size n with replacement from our data

2. compute our statistic
3. do 1. & 2. many times

The resulting distribution of statistics across our resamples is an

approximation of the sampling distribution of our statistic
The idea is that a random sample of a random sample from the
population, is again a random sample of the population
This is called the bootstrap and computes the sampling distribution
without a model!

41/45
Approximation through numerical simulation

n = 1000
data = rbinom(n, 1, .54) # true distr, usually unknown
estimates = rep(0,999)
for(i in 1:999) {
id = sample(1:n, n, replace=T)
estimates[i] = mean(data[id])
}
sd(estimates)

## [1] 0.015946413

sqrt(.54*(1-.54)/1000) # true value, usually unknown

## [1] 0.015760711
42/45
Summary

Probability

Population Sample (Data)

fX (x ) (x1 , . . . , xn )

Parameter Statistic
x̄ = n1 ni=1 xi
P
E [X ] = µ

Inference

43/45
Summary

Today we looked at the elements of statistical inference

I Estimation:
I Determining the distribution, or some characteristic of it.
(What is our best guess for p?)
I Confidence intervals:
I Quantifying the uncertainty of our estimate. (What is a range
of values to which we’re reasonably sure p belongs?)
I Hypothesis testing:
I Asking a binary question about the distribution. (Is p > 0.5?)

44/45
Summary

In statistical inference we think of data as a realization of a random

process
There are many reasons why we think of our data as (ex-ante)
random:

1. We introduced randomness in our data collection (random

sampling, or random assigning treatment)
2. We are actually studying a random phenomenon (coin tosses or
dice rolls)
3. We treat as random the part of our data that we don’t
understand (errors in measurements)

The coming weeks we will take a closer look at how this randomness
affects what we can learn about the population from the data

45/45