Statistical Methods

Never trust a statistics you didnt forge yourself

Winston Churchill

Florian Herzog

2013
 Independent and identical distributed random variables

Definition 1. The random variables X1, ..., Xn are called a random sample
of size n from the population f (x) if X1, ..., Xn are mutually independent
random variables and the marginal pdf of each Xi is the same function Xi.
Alternatively, X1, ..., Xn are called independent and identically distributed
random variables with pdf f (x).

The joint pdf of X1, ..., Xn is given as:

n
Y
f (x1, x2, ..., xn) = f (x1)f (x2) f (xn) = f (xi)
i=1

The slides of this section follow closely the Chapters 5 and 7 of the Book
[Link] and R. Berger, Statistical Inference,Duxbury Press 2002.

Stochastic Systems, 2013 2

 Identically and independently distributed random variables

Often in statistics (especially in estimation) we assume identically and inde-

pendently distributed (i.i.d) random variables (r.v.). This means that a random
variable Xi, where k = 1, 2, ... denotes the realizations of the r.v., has the
following properties:
Each Xk f (x) is drawn form the same density .
Xk is independent of Xk1, Xk2, ..., X1.
Each Xk is uncorrelated from each Xj ,i.e. E[Xk Xj ] = 0 j\{k}

Stochastic Systems, 2013 3

 Sample mean and variance

Definition 2. The sample mean is the arithmetic average of the values in a

random sample and is denoted as
n
1X
X = Xi
n i=1

Definition 3. The sample variance is the statistic defined as

n
2 1 X 2
S = (Xi X)
n 1 i=1

(S 2)
p
The sample standard deviation is the statistic defined as S =

Stochastic Systems, 2013 4

 Properties of [Link] variables

Theorem 1. When we have X1, X2, ...Xn independent and identically

distributed (i.i.d) random variables with mean = E[Xn] and variance
2 = V ar[Xn]. Then

E[X] =
2
V ar[X] =
n
2 2
E[S ] =

Stochastic Systems, 2013 5

 Properties of [Link] variables

Theorem 2. When we have X1, X2, ...Xn i.i.d. from a normal distribution
with mean and variance . Then
1. X and S 2 are independent random variable,
2
2. X is distributed N (, n ),
2
3. (n 1) S2 has a chi square distribution with n 1 degrees of freedom.

Stochastic Systems, 2013 6

 Convergence of a sequence of r.v. {Xn}

1
1. Convergence with probability one (or almost sure), Xn X :

P ({ : lim (Xn()) = X()}) = 1.

n

p
2. The sequence {Xn} converges to X in probability, Xn X , if
 
lim P ({ : |Xn() X()| > }) = 0, for all > 0.
n

p Lp
3. The sequence {Xn} converges to X in L , Xn X , if
 
p
lim E(|Xn() X()| ) = 0.
n

Stochastic Systems, 2013 7

 Convergence concepts interrelations

Lp
Xn X
?
1 Lq
Xn X (almost sure) Xn X, q<p
q )
p
Xn X (in probability)

?
d
Xn X (in distribution)

Stochastic Systems, 2013 8

 Parameter estimation (Point estimation)

In stochastic systems modeling, we often build models from data observation

(and not from physical first principles).
We need statistically motivated methods to identify the stochastic systems
under consideration. The identification of the stochastic systems requires the
following:
Identification of the distribution
Identification of the dynamics
Identification of the system parameters
Analysis of the parameter significance
In this section we only focus on the parameter estimation and assume that
the distribution is known. We will come back to this topic after the theoretical
introduction of stochastic processes.

Stochastic Systems, 2013 9

 Parameter estimation (Point estimation)

Definition 1. A point estimator is any function W (X1, X2, ..., Xn) of a

sample of random variables.

There are main ways of finding point estimators, the main ones are:
Methods of moments (MM)
Maximum Likelihood estimators (MLE)
Expectation Maximization (EM)
Bayes Estimators
Besides the methods of finding a point estimator, the evaluation (quality) of the
estimator. In the following slides, we will introduce the methods of moments
and the maximum likelihood estimator.

Stochastic Systems, 2013 10

 Methods of moments

We have X1, X2, ...., Xn the sample from a population from one pdf
f (x|1, 2, ..., k ). The parameter i are the distribution parameter, e.g.
and in the case of a normal distribution.

Definition 2. The method of moments is the matching of the first k moments

of the data with the first k theoretical moments of the distribution. The
theoretical moments are a function of the parameters and parameter estimation
problem is reduced to the solving of k equations

Stochastic Systems, 2013 11

 Methods of moments

We have
n
1X 0
m1 = Xi, 1 = E[X],
n i=1
n
1X 2 0 2
m2 = Xi , 2 = E[X ],
n i=1
...
n
1X k 0 k
mk = Xi , k = E[X ],
n i=1

where mi denotes the sample moment and 0i the theoretical moments.

Stochastic Systems, 2013 12

 Methods of moments

Since 0i is a function of i, we get the following system of equations:

0
m1 = 1(1, ..., k ),
0
m2 = 2(1, ..., k ),
...
0
mk = k (1, ..., k ),

where mi denotes the sample moment and 0i the theoretical moments.

The parameters are found by solving the system of k moments.

Stochastic Systems, 2013 13

 Methods of moments

As main example, we assume that the data is generated by a normal distribution

with mean and variance 2. We denote 1 = and 2 = 2. The first and
second moment of the normal distribution are given as
n
0 1X
1 = = Xi
n i=1
n
0 2 1X 2
2
2 = + = Xi
n i=1

Stochastic Systems, 2013 14

 Methods of moments

Solving for and 2 we get:

n
1X
= Xi
n i=1
n n
!2 n
2 1X 2 X 1X 2
= Xi Xi = (Xi )
n i=1 i=1
n i=1

The solution are the sample moments of mean and variance and are of course
the naturalway of estimation the mean and variance of the normal distribution.

Stochastic Systems, 2013 15

 Maximum Likelihood Estimation

The likelihood is the joint pdf of X1, ..., Xn and given as:
n
Y
L(1, ...k |x1, x2, ..., xn) = f (xi|1, ...k ) .
i=1

We denote by x = [x1, x2, ...]T and by = [1, 2, ...]T

Definition 3. For each sample x, let (x) b be a parameter value at which

L(|x) attains its maximum as function of . A maximum likelihood estimator
MLE of the parameter based on the X is (x) b .
If the likelihood function is C 2, then possible candidates for the MLE are the
values of which solve

L(1, ...k |x1, x2, ..., xn) = 0 .
i

Stochastic Systems, 2013 16

 Maximum log-Likelihood Estimation

Theorem 1. The maximum likelihood estimation is equivalent to the maxi-

mum log-likelihood estimation. The log-likelihood is defined as
n
X
l(1, ...k |x1, x2, ..., xn) = log (f (xi|1, ...k )) .
i=1

Example: We want to derive the maximum likelihood estimated for the mean
() of the normal distribution under the assumption of known variance 2. The
log-pdf of the normal distribution is given as:
!
2
1 2 (xi )
log(f (xi|)) = log(2 ) +
2 2

Stochastic Systems, 2013 17

 Maximum log-Likelihood Estimation

The maximum log-likelihood function is given as:

n
!
2
X 1 2 (xi )
l(|x1, x2, ..., xn) = log(2 ) +
i=1
2 2

Since is known, the maximization problem is reduced to least square problem:

n
X 2
min (xi )

i=1

which as the solution of

n
1X

b= xi
n i=1

Stochastic Systems, 2013 18

 Invariance of Maximum Likelihood Estimation

Theorem 2. The invariance property of MLEs state that if b is the MLE of

then for any function (), the MLE of () is ()
b.

Suppose that a distribution is parameterized by a parameter , but we are

interest in finding an estimator for some function of , say (), then we can
still use the MLE for . An example is as follows: If is the mean of normal
distribution, the MLE of sin() is sin(b).

Stochastic Systems, 2013 19

 Quality of estimators: MSE

Definition 4. The mean squared error (MSE) of an W of the a parameter

is the function defined by E [(W )2.
The MSE of W measures the average squared distance between the estimator
and the true value of the parameter. The MSE has the following interpretation:
2 2
E [(W ) ] = V ar [W ] + (E [W ] )

The first term is the variance of the estimator W and the second term is called
the bias.
Definition 5. The bias of an estimator W of the parameter is the distance
between the expected value of W and the true value of . An estimator where
the bias is zero is called an unbiased estimator.

Stochastic Systems, 2013 20

 Quality of estimators: Bias and variance

In the multivariate case where is a vector of parameters, the variance of the

estimator is a covariance Cov(). An estimator with low variance (covariance)
is called an efficient estimator (in the sense that few data is needed). The MSE
is often an trade-off between unbiasedness and higher variance or an biased but
efficient estimator.
The true value of the MSE can often not be determined since the true value of
is not known. Therefore, we focus on the variance of the estimator in order
to describe the quality of the estimator.
p p
Definition 6. An estimator is called called consistent when b where
denotes convergence in probability. An unbiased estimator is also consistent.

Stochastic Systems, 2013 21

 Quality of estimators: Normal distribution example

When we have X1, X2, .. i.i.d. data from a N (, 2) distribution and use the
sample mean X and sample variance S 2 as estimator:
E[X] = and therefore, X is unbiased
E[S 2] = 2 and therefore, S 2 is unbiased
2
E[(X )2] = V ar[X] = n
2 4
E[(S 2 2)2] = V ar[S 2] = n1
2
The MSE of X is still n when the data is not normal, but this does not hold
for MSE for S 2 when the data is not normally distributed.

Stochastic Systems, 2013 22

 Quality of estimators: Cramer-Rao bound for the variance

Definition 7. The Fisher Information matrix J is defined as


1
Ji,j = l(1, ...k |x1, x2, ..., xn)
n i


l(1, ...k |x1, x2, ..., xn) ,
j

which is known as the outer product form. Under certainty regularity conditions
and when the log-likelihood function is C 2, it can be calculated as:
!
2
1
Ji,j = l(1, ...k |x1, x2, ..., xn) ,
n ij

which is called the inner product form. Note that the expectation is conditional
on

Stochastic Systems, 2013 23

 Quality of estimators: Cramer-Rao bound for the variance

The Cramer-Rao bound states the following:

Theorem 3. The covariance of an estimator W is bounded by

J 1
Cov(W ) ,
N
where N is the number of observations- This bound also to make an worst case
approximation of the efficiency of an estimator.
The Fisher Information matrix allows us to compute the uncertainty and thus,
the quality of an estimator.

Stochastic Systems, 2013 24

 Quality of estimators

MLE is the main methods for finding estimators, since it has the following
properties:
Consistency: the estimator converges in probability to the value being
estimated.
Asymptotic normality: as the sample size increases, the distribution of the
MLE tends to the Gaussian distribution with mean and covariance matrix
equal to the inverse of the Fisher information matrix.
Efficiency, i.e., it achieves the Cramer-Rao lower bound when the sample
size tends to infinity. This means that no asymptotically unbiased estimator
has lower asymptotic mean squared error than the MLE
1
The estimate of N (M L, JN ).
Barretts Theorem The maximum-likelihood procedure in any problem is
what you are most likely to do if you dont know any statistics.

Stochastic Systems, 2013 25