Probability and statistics

Lecture notes

Thanos Mergoupis

9.3.25

Part 2.1: Random sampling and

sampling distributions
We have seen that the investigation into how random variables relate to each other

often focuses on how features of conditional distributions change as the values of the

conditioning variable(s) change. If the joint distribution of two random variables is

known, we have developed tools to carry out such an investigation. More often

however, we do not know the exact joint distribution of the variables we are interested

in. We then have to infer this joint distribution using pieces of information drawn

from it. Statistical inference gives us the tools to do this.

We start the study of statistical inference by studying how pieces of

information or samples from a distribution, are related to the distribution they are

drawn from. To do this, we initially assume that we know the distribution or

population from which these pieces of information, or samples, are drawn from.

Random samples

Definition:

Let X1, X2, X3, …, Xn represent independent drawings from the distribution, or

population, of the random variable X. The ordered set {X1, X2, X3, …, Xn} is called a

random sample of size n on the random variable X.

1
Note

Every Xi for i=1, 2, 3, …., n is a r.v. because its values are determined by the random

experiment of drawing a value from the distribution of X. In fact, the likelihood of

each value of the Xi equals the likelihood of the same value of X, since the draws are

from the distribution of X. Therefore the distribution of each Xi is identical to the

distribution of X. Moreover, because the random draws are independent from each

other, the Xi are distributed independently of each other. Therefore the Xi are

independently and identically distributed or i.i.d. When a specific sample is drawn, we

have a realisation of this set of variables.

Because a random sample is an ordered set, we can represent it using vector notation.

We use bold to denote vectors so that X denotes the random vector (X1, X2, X3, …, Xn)

and x denotes the realisation of this random vector, i.e. (x1, x2, x3, …, xn).

Because by definition the Xi are identically and independently distributed, if they are

drawn from the distribution f(x) of a random variable X, their joint distribution gn(x) is

given by:

gn (x) = gn ( x1 , x2 ,..., xn ) = f1 ( x1 )  f 2 ( x2 )  ...  f n ( xn ) =

The first line uses the fact that the Xi are independently distributed, so that their joint

distribution equals the product of the marginal distributions. The second line uses the

2
fact that the Xi are all identically distributed with their distribution identical to the

distribution they are drawn from.

Example

The exponential parametric family is a one-parameter family of distributions. A

random variable X follows an exponential distribution with parameter λ if its pdf is

given by:

e− x for x  0
f ( x) = 
0 otherwise

Then a random sample size n from this distribution has joint probability density

function given by:

g n (x) = f ( x1 )  f ( x2 )  ...  f ( xn ) =  e −  x1   e −  x2  ...   e −  xn =

for x with all xi positive, and gn (x) = 0 otherwise.

Sample statistics

Let T = h(X1, X2, X3, …, Xn) = h(X). That is, the values of T are determined by a

function h(  ) of the random sample. Then T is called sample statistic.

Because sample statistics are functions of random vectors, they are random variables

themselves. Their values are determined by the n independent draws of a random

sample.

Examples of sample statistics

Sample Mean

3
The sample mean is defined as the arithmetic average of a random sample. That is:

1 n
X = ( X1 + X 2 + X 3 + ... + X n ) / n =  Xi
n i =1

Sample Variance

The sample variance is defined as:

1 n
S2 =  ( X i − X )2
n − 1 i =1

Examples of other sample statistics are:

Sample maximum

Xmax = max(X1, X2, X3, …, Xn)

Sample minimum

Xmin = min(X1, X2, X3, …, Xn)

Sample range

Xmax-Xmin

Sample midrange

(Xmax+Xmin)/2

Sampling distributions

The distribution of a sample statistic is called sampling distribution. In general, a

sampling distribution depends on:

4
1. The function h( , , , …, ) that determines the values of the sample statistic.

2. The distribution the sample is drawn from, i.e. the pmf or the pdf f(X).

3. The size n of the random sample.

The derivation of sampling distributions has variable degrees of difficulty. Certain

features of sampling distributions however, can be derived quite easily.

The sampling distribution of the sample mean

The sample mean theorem

Given a random sample size n { X1, X2, X3, …, Xn} from a population with E(X) = ,

and V(X) = 2, the sampling distribution of the sample mean has:

E( X ) = 

V (X ) =  2 / n

That is, the expected value of the sample mean is equal to the expected value of the

population from which the sample was drawn, and the variance of the sample mean is

equal to the variance of the population divided by the sample size.

Note that the mean of the sampling distribution of the sample mean does not vary

with the sample size, but the variance of the sample mean does. As sample size

increases, the distribution of the sample mean becomes more concentrated about its

mean.

When sampling from some distributions we can have more precise results on the

sampling distribution of the sample mean.

5
Sampling from the normal distribution

When sampling, with sample size n, on the r.v. X with X ~ N(,2), then:

X ~ N (, 2 / n)

The fact the sample mean has mean μ and variance σ2/n follows directly from the

sample mean theorem. The fact that the sample mean is also a normal random

variable follows from the property that linear combinations of independent normal

random variables are also normally distributed.

Example

In a British city the (natural) logarithm of annual family income before taxes follows

a normal distribution with mean 9.680 and variance 9.105. We draw a random sample

of 10 families from this population and ask for their family income. What is the

probability that the mean income of this sample is greater than £26,000?

Let Y = log(annual family income before taxes)

Then it is given that Y ~ N(9.680, 9.105)

We have that log(26,000)=10.166

Because the population the sample was drawn from is normal, the sample mean is

distributed as:

Y ~ N (9.680,9.105/10)  Y ~ N (9.680,0.9105)

Then:

 Y − 9.680 10.166 − 9.680 

6
=1 – P( Z  0.5093) = 1 – 0.695 = 0.305

Clearly it is necessary to know the exact distribution of a sample mean in order to

calculate probabilities like the one in this example. As suggested earlier, although the

sample mean is a fairly simple function of the random sample, its distribution will

vary with the population sampled. The distributions of sample means from some well

known parametric families are known, but in general one would have to work it out.

This is true if we want to evaluate probabilities of a sample mean exactly. If, however,

we are prepared to accept a small margin of error, then there is a remarkable result

that identifies the distribution of any sample mean, regardless of the population

sampled, as long as we have a large enough sample. This is the Central Limit

Theorem.

Central Limit Theorem

In random sampling size n on any random variable X with E(X) = , and V(X) = 2 ,

as the size of a random sample increases, the sampling distribution of the sample

mean approaches (in some sense we have not defined) a normal distribution. In

particular, the standardised sample mean

X −
Z=
/ n

approaches the standard normal distribution N(0,1).

7
The use of the expression “approaches in some sense” is vague, but it refers to

something well defined mathematically. The sense in which “approaches” is used here

is called “convergence in distribution” but we do not need to define this concept here.

Example

Suppose it is known that the average distance a student travels to the University of

Bath is 2 miles, with standard deviation 1.2 miles. You survey a random sample of 50

students on the distance they travelled to get to the University. What is the probability

that the sample mean will be at most 1.75 miles?

Here we do not know the exact population we are sampling from. The only things we

know are that it has a mean of 2 miles and a standard deviation of 1.2 miles. But

because the sample is somewhat large, we can appeal to the CLT. So:

Let X = distance travelled to the University of Bath.

Then:

 X − 2.0 1.75 − 2.0 