CL205: AI & DS

MB
Mani Bhushan
Department of Chemical Engineering,
Indian Institute of Technology Bombay
Mumbai, India- 400076
mbhushan@[Link]

Ack: Prof. Sachin Patwardhan

Autumn 2023

MB (IIT Bombay) CL205 Autumn 2023 1 / 40

Today’s lecture:

Sampling Statistics and their Distributions

Chapter 6 of Ross

MB (IIT Bombay) CL205 Autumn 2023 2 / 40

Probability and Statistics (1)

So far, looked at probability.

I Start with a “model” (such as probability density function or
distribution function) describing what events we think will occur
and with what likelihoods.
I Typically about prediction: looking forward. Predict what data
will we observe from given rules.
I Probability is mathematics: no ambiguity.
Statistics
I Statistics typically about looking backwards: extract rules
(unknown parameters) from given data.
I An art to some extent: how to sumamrize data (mean/median)
or visualize data (histogram, bar plot, pie chart, etc.), what is a
good hypothesis to test at what significance level, etc.

MB (IIT Bombay) CL205 Autumn 2023 3 / 40

Probability and Statistics (2)

Probability and Statistics are linked

I Use of statistics to test hypotheses on population parameters,
extract relationships, etc.
I Use of the extracted models to predict future.
I Observe data (new statistics), refine probability models,
continue

MB (IIT Bombay) CL205 Autumn 2023 4 / 40

Statistics, Population and Sample

Statistics deals with drawing conclusions from observed data.

Population: Large (infinite) collection of items that have
measurable values associated with them.
Sample: A finite collection of items from the population that is
observed.
Problem: Use sample to draw inferences about population.

MB (IIT Bombay) CL205 Autumn 2023 5 / 40

Populations and samples

We sample to draw some conclusions about an entire population.

Examples: Exit polls, census.
We assume that the population can be described by a probability
distribution.
We assume that the samples give us measurements following
this distribution.

MB (IIT Bombay) CL205 Autumn 2023 6 / 40

Formally...

If X1 , ..., Xn are independent random variables having a common

distribution F i.e. X1 , ..., Xn are IID (independently and
identically distributed), then we say that they constitute a
sample (or a random sample) from the distribution F .
In most applications, F not completely known and problem is to
use samples to infer F .
Parameteric inference: F specified in terms of unknown
parameters, such as height is gaussian with unknown mean and
variance,
Non-parameteric inference: nothing known about the form of F .
Parameteric inference in this course.

MB (IIT Bombay) CL205 Autumn 2023 7 / 40

Distributions and Parameteric Estimation

We wish to infer the parameters of a population distribution.

I mean, variance, proportion etc.
We use a statistic derived from the sample data.
I Sample mean, sample variance etc.
A statistic is a random variable.
I What distribution does it follow?
I What are the parameters of its distribution?

MB (IIT Bombay) CL205 Autumn 2023 8 / 40

Notation

Population parameters are typically in Greek.

I µ = population mean.
I σ 2 = population variance.
The corresponding statistic is not in Greek
I X̄ = sample mean.
I S 2 = sample variance.
We can also denote an estimate of the parameter θ as θ̂.
Regression:
I You want: y = αx + β.
I You get: y = ax + b.

MB (IIT Bombay) CL205 Autumn 2023 9 / 40

What can we say about Sample Mean
Consider a population with mean µ and variance σ 2 .
We have n random samples X1 , X2 , . . ., Xn from this population.
Define sample mean as:
X1 + · · · + Xn
X̄ =
n

X̄ is a random variable since it is a function of the random

variables X1 , ..., Xn .
What do we expect from X̄ ?
E [Xi ] = µ var (Xi ) = σ 2

 
X1 + · · · + Xn E [X1 ] + · · · + E [Xn ]
E [X̄ ] = E =
n n
nµ
= =µ
n
MB (IIT Bombay) CL205 Autumn 2023 10 / 40
Why do we infer µ using the sample mean?

E [X̄ ] = µ

X̄ is an unbiased estimator of µ.
For a statistic θ̂ to be an unbiased estimator of θ,

E [θ̂] = θ

MB (IIT Bombay) CL205 Autumn 2023 11 / 40

What is the variance of X̄ ?

We had var (Xi ) = σ 2 .

 
X1 + · · · + Xn
var (X̄ ) = var
n
1
= 2 (var (X1 + · · · + Xn ))
n
1
= 2 (var (X1 ) + · · · + var (Xn )) Independence!
n
1
= 2 nσ 2
n
σ2
=
n

MB (IIT Bombay) CL205 Autumn 2023 12 / 40

Xi versus X̄ ?

E [Xi ] = µ var (Xi ) = σ 2

σ2
E [X̄ ] = µ var (X̄ ) =
n

Variance of sample mean decreases with increasing n.

Sample more and gain accuracy when using X̄ as an estimate of
µ.
What can we say about distribution of sample mean?

MB (IIT Bombay) CL205 Autumn 2023 13 / 40

The Central Limit Theorem

Theorem
Let X1 , X2 , ..., Xn be a sequence of independent and identically
distributed random variables each having mean µ and variance σ 2 .
Then for large n, the distribution of
X1 + X2 + .... + Xn
is approximately normal with mean nµ and variance nσ 2 .

i.e. for large n

(X1 + · · · + Xn ) ∼ N (nµ, nσ 2 )
 
(X1 + · · · + Xn ) − nµ
P √ <x ≈ P(Z < x)
σ n
with Z being a standard normal random variable.

MB (IIT Bombay) CL205 Autumn 2023 14 / 40

Central Limit Theorem: History

One of the most powerful results in probability

Proposed in 1733 by French mathematician A. deMoivre.
Forgotten until Laplace published it in 1812: used normal to
approximate binomial.
In 1901 Lyapunov defined it in general terms and proved it
formally.

MB (IIT Bombay) CL205 Autumn 2023 15 / 40

The Central Limit Theorem
Consider the total obtained on rolling several dice.

MB (IIT Bombay) CL205 Autumn 2023 16 / 40

The Central Limit Theorem
Consider the total obtained on rolling several dice.

MB (IIT Bombay) CL205 Autumn 2023 17 / 40

The Central Limit Theorem
Consider the total obtained on rolling several dice.

MB (IIT Bombay) CL205 Autumn 2023 18 / 40

The Central Limit Theorem
Consider the total obtained on rolling several dice.

MB (IIT Bombay) CL205 Autumn 2023 19 / 40

The Central Limit Theorem
Consider the total obtained on rolling several dice.

MB (IIT Bombay) CL205 Autumn 2023 20 / 40

Galton’s Box

Figure: From Wikipedia (Matemateca (IME/USP)/Rodrigo Tetsuo

Argenton)

MB (IIT Bombay) CL205 Autumn 2023 21 / 40

Central Limit Theorem (CLT) and the Binomial
Let X = X1 + · · · + Xn , where Xi is a Bernoulli variable (1 if
success and 0 if failure; probability of success p).

E [Xi ] = p var (Xi ) = p(1 − p)

E [X ] = np var (X ) = np(1 − p)

CLT suggests that for large n

X − np
p ∼ N (0, 1)
np(1 − p)

Rule of thumb: The Binomial can be approximated well with a

normal if
np(1 − p) ≥ 10
MB (IIT Bombay) CL205 Autumn 2023 22 / 40
Example 6.3c

Q Ideal size of a first year class at a college is 150 students. The

college, knowing from past experience that on the average only
30% of those accepted for admission will actually attend, uses a
policy of approving the applications of 450 students. Compute
the probability that more than 150 first year students attend this
college.
A X is the number of students that attend. Then X is a binomial
RV with n = 450 and p = 0.3 assuming each student
independently decides whether to attend or not.
Note X (binomial) is discrete while normal is continuous. Thus

P(X = i) = P(i − 0.5 < X < i + 0.5)

MB (IIT Bombay) CL205 Autumn 2023 23 / 40

Example (continued)

Thus,
!
X − (450)(0.3) 150.5 − (450)(0.3
P(X > 150.5) = P p ≥ p
(450)(0.3)(0.7) (450)(0.3)(0.7)

≈ P(Z ≥ 1.59) = 0.06

MB (IIT Bombay) CL205 Autumn 2023 24 / 40

Approximate Distribution of Sample Mean
Q Let X1 , ..., Xn be a sample from a population having mean µ and
variance σ 2 . What is the distribution of sample mean?
Pn
Xi
X̄ = i=1
n

A Using CLT, distribution of ni=1 Xi is approximately normal

P
when n is large.
Then distribution of X̄ is also normal since constant multiple of
a normal RV is also normal RV, with
E [X̄ ] = µ; var (X̄ ) = σ 2 /n
Hence,
X̄ − µ
√
σ/ n
has a standard normal distribution.
MB (IIT Bombay) CL205 Autumn 2023 25 / 40
CLT and Sample Size for X̄ to be Normal

CLT does not tell us how large sample size n needs to be for the
normal approximation of X̄ to be valid.
n depends on population distribution of the sample data.
For binomial np(1 − p) ≥ 10, for normal any n ≥ 1 is ok.
Rule of thumb: sample size n ≥ 30 works for almost all
distributions, i.e. no matter how nonnormal the underlying
population is, the sample mean of a sample of size atleast 30
will be approximately normal.
In most cases, normal approximation will be valid for much
smaller sample sizes.

MB (IIT Bombay) CL205 Autumn 2023 26 / 40

Densities of Sample Means of a Normal Population
 
1
If X ∼ N (0, 1), then X̄ ∼ N 0,
n

MB (IIT Bombay) CL205 Autumn 2023 27 / 40

Sample variance

Let X1 , X2 , ..., Xn be a random sample from a distribution with

mean µ and variance σ 2 .
Let X̄ be the sample mean.
The statistic S 2 defined as:
Pn
2 (Xi − X̄ )2
S = i=1
n−1
is called the sample variance. It is a random variable.
√
S = S 2 is called the sample standard deviation.

MB (IIT Bombay) CL205 Autumn 2023 28 / 40

E [S 2]?
We compute E [S 2 ] as follows:
(Xi − X̄ )2 = (Xi − µ + µ − X̄ )2
= (Xi − µ)2 + (µ − X̄ )2 + 2(Xi − µ)(µ − X̄ )
n
X Xn Xn
(Xi − X̄ )2 = (Xi − µ)2 + (µ − X̄ )2
i=1 i=1 i=1
n
X
+2 (Xi − µ)(µ − X̄ )
i=1

The middle term is ...

n
X
(µ − X̄ )2 = n(µ − X̄ )2
i=1

MB (IIT Bombay) CL205 Autumn 2023 29 / 40

E [S 2] (Continued)

The last term is ...

n
X n
X
2 (Xi − µ)(µ − X̄ ) = − 2(X̄ − µ) (Xi − µ)
i=1 i=1
2
= − 2n(X̄ − µ)
n
X n
X
(Xi − X̄ )2 = (Xi − µ)2 + n(µ − X̄ )2 − 2n(X̄ − µ)2
i=1 i=1
n
X
= (Xi − µ)2 − n(X̄ − µ)2
i=1
Pn 2
Pn
− X̄ )
i=1 (Xi i=1 (Xi− µ)2 n(X̄ − µ)2
= −
n−1 n−1 n−1

MB (IIT Bombay) CL205 Autumn 2023 30 / 40

E [S 2] (Continued)
We had
Pn Pn
2 − X̄ )2
i=1 (Xi i=1 (Xi− µ)2 n
S = = − (X̄ − µ)2
n−1 n−1 n−1

Taking expectations
Pn 2
 Pn 2

2 i=1 (Xi − X̄ ) i=1 (Xi − µ)
E [S ] =E =E −
n−1 n−1
 
n 2
E (X̄ − µ)
n−1

But,
Pn 2
σ2

i=1 (Xi − µ) n
E = σ 2 , E [(X̄ − µ)2 ] = var (X̄ ) =
n−1 n−1 n
MB (IIT Bombay) CL205 Autumn 2023 31 / 40
E [S 2] (continued)

This implies
Pn 2
σ2

i=1 (Xi − X̄ ) n
E = σ2 −
n−1 n−1 n−1
2
= σ

or
E [S 2 ] = σ 2

Therefore S 2 is an unbiased estimator of σ 2 .

This is the real reason for having n − 1 in the denominator in the
S 2 expression instead of n, and not really the degrees of freedom
argument given earlier.

MB (IIT Bombay) CL205 Autumn 2023 32 / 40

Sampling Distributions from a Normal Population

Consider distribution of statistics (X̄ , S 2 ) obtained from samples from

a Normal population.
Theorem
If X1 , X2 , ..., Xn is a sample (IID) from a normal population having
mean µ and variance σ 2 , then X̄ and S 2 are independent random
variables, with X̄ being normal with mean µ and variance σ 2 /n and
(n − 1)S 2 /σ 2 being chi-square random variable with n − 1 degrees of
freedom.

X̄ is approximately normal due to CLT in general. However in

this case, it is exactly normal since sum of normal RVs is normal

MB (IIT Bombay) CL205 Autumn 2023 33 / 40

Sampling Distributions from Normal Population
(Cont.)

(n − 1)S 2 /σ 2 is a chisquare with n − 1 degrees of freedom.

X̄ , S 2 are independent random variables.
Proof (non-rigourous) of last two statements in Ross.
Independence of X̄ and S 2 due to normal population, not true in
general.

MB (IIT Bombay) CL205 Autumn 2023 34 / 40

The Chi-Square Distribution
If Z1 , Z2 , ..., Zn are independent standard normal random
variables, then X , defined as
X = Z12 + Z22 + ... + Zn2
is said to have a chi-square distribution with n degrees of
freedom or
X ∼ χ2n

Let X1 , X2 be independent chi-square random variables with n1

and n2 degrees of freedom, respectively. Then X1 + X2 is
chi-square with n1 + n2 degrees of freedom.
For X a chi-square RV with n degrees of freedom, quantity χ2α,n
is defined such that
P(X ≥ χ2α,n ) = α

MB (IIT Bombay) CL205 2 Autumn 2023 35 / 40

Chi-squared PDF Sketch

MB (IIT Bombay) CL205 Autumn 2023 36 / 40

Chi-squared RV

Density function of chi-squared RV involves gamma function.

(without proof) For a chi-squared RV with n degrees of freedom:
E [X ] = n, Var(X ) = 2n.

MB (IIT Bombay) CL205 Autumn 2023 37 / 40

Example 6.5a to illustrate use of S 2 distribution

Q The time it takes for a processor to carry out a particular

computation is normally distributed with mean 20 seconds and
standard deviation 3 seconds. If a sample of 15 such
computations is observed, what is the probability that the
sample variance is greater than 12?
A n = 15, σ 2 = 32 = 9. Then, P(S 2 > 12) =
 
14
P 14S /9 > 12 = P(χ214 > 18.67)
2
9

MB (IIT Bombay) CL205 Autumn 2023 38 / 40

Summary

For any sample: E [X̄ ] = µ, E [S 2 ] = σ 2

CLT: sum of IID RVs is approximately normal. As a result, X̄ is
approximately normal with mean µ and variance σ 2 /n.
When sample from normal population, then X̄ is exactly normal
and (n − 1)S 2 /σ 2 is a chi-square RV with n − 1 degrees of
freedom. Further, X̄ , S 2 are independent.

MB (IIT Bombay) CL205 Autumn 2023 39 / 40

 THANK YOU

MB (IIT Bombay) CL205 Autumn 2023 40 / 40