Stat425 (F16) F.

Liang

The Univariate Normal Distribution

• Z ∼ N(0, 1) is called the standard normal rv. Φ(z) denotes its CDF.

Φ(−z) = 1 − Φ(z), z > 0.

• X ∼ N(µ, σ 2 ), then

• X = µ + σZN(µ, σ 2 ). pdf, mgf, mean and variance.

1
• aX + b ∼ N(aµ + b, a2 σ 2 ). Specially, σ (X − µ) ∼ N (0, 1).

• Linear combinations of independent normal rv’s are normal.

• Zi iid ∼ N(0, 1). What’s the distribution of W = Z12 + · · · + Zn2 ?

n
2
Y
MW (t) = EetW = EetZi = (1 − 2t)−n/2 .
i=1

So W ∼ Ga( n2 , 12 ) = χ2n .

E(W ) = n, Var(W ) = 2n.

( 21 )n/2 n −1 − w
fW (w) = w2 e 2 , w > 0.
Γ( n2 )

• Z ∼ N(0, 1) and W ∼ χ2n are independent, then √ Z ∼ Tn (student t-dist with n

W/n
degrees of freedom).

How to derive the pdf of T = √ Z ? Consider the following bivariate transformation

W/n

Z
T =p , V = W.
W/n

Then
− n+1
Γ( n+1 ) t2
Z  2
fT (t) = f (t, v)dv = √ 2 n 1+ .
nπΓ( 2 ) n

1
Stat425 (F16) F. Liang

The Bivariate Normal Distribution

(Y1 , Y2 ) ∼ N2 (µ, Σ), where

σ12 σ12 σ12

     
µ1 ρσ1 σ2
µ= , Σ= = ,
µ2 σ12 σ22 ρσ1 σ2 σ22

• µ1 and σ12 are the mean and variance, respectively, of Y1 ;

• µ2 and σ22 are the mean and variance, respectively, of Y2 ;

• σ12 = ρσ1 σ2 is the covariance of Y1 and Y2 with ρ being the correlation coefficient.

Assume ρ2 < 1, and the joint pdf f (y1 , y2 ) is given by

1
p ×
2πσ1 σ2 1 − ρ2
   y − µ  y − µ   y − µ 2 
1 y1 − µ1 2 1 1 2 2 2 2
exp − − 2ρ +
2(1 − ρ2 ) σ1 σ1 σ2 σ2
 
1 1 t −1
= exp − (y − µ) Σ (y − µ) .
2π|Σ|1/2 2

Note that |Σ| = σ12 σ22 (1 − ρ2 ) and when ρ2 < 1,

σ22
 
−1 1 −ρσ1 σ2
Σ = 2 2 .
σ1 σ2 (1 − ρ2 ) −ρσ1 σ2 σ12

• Marginals: Yi ∼ N(µi , σi2 ) where i = 1, 2.

• Conditionals:
 σ1 
Y1 | Y2 = y2 ∼ N µ1 + ρ (y2 − µ2 ), (1 − ρ2 )σ12
σ2
 σ2 
Y2 | Y1 = y1 ∼ N µ2 + ρ (y1 − µ1 ), (1 − ρ2 )σ22
σ1
• Linear Combinations:

aY1 + bY2 ∼ N aµ1 + bµ2 , a2 σ12 + 2abρσ1 σ2 + b2 σ22

• Uncorrelated = Independent: ρ = 0 implies Y1 and Y2 are independent.

• Note: All the statements above assume that the joint distribution of (Y1 , Y2 ) is normal.
However,

Y1 ∼ Norm, Y2 ∼ Norm does NOT imply (Y1 , Y2 ) ∼ Norm.

So if Y1 and Y2 are marginally normally distributed with correlation 0, we cannot

conclude that Y1 and Y2 are independent.

2
Stat425 (F16) F. Liang

The Multivariate Normal Distribution

• Let Z = (Z1 , . . . , Zn ) where Zi ’s are iid ∼ N(0, 1) rv’s. Then Z follows a multivariate
normal distribution, denoted by Nn (0, In ), with
n n
Y 1 −z 2 1 n 1X
2
o 1 n 1 o
f (z) = √ e i = n/2
exp − z i = n/2
exp − zt z ,
2π (2π) 2 (2π) 2
i=1 i=1

n
Y n1 o
M (t) = E[exp{tt Z}] = Eeti Zi = exp ktk2 ,
2
i=1
and
E(Z) = 0, Cov(Z) = In .

• We can define a general multivariate normal distribution via affine transformations.

Xp×1 = µp×1 + Bp×n Zn×1 ,

then X is multivariate normal with mean µ and covariance matrix Σ = BB t , denoted

by
X ∼ Np (µ, Σ),
with n 1 o
MX (t) = exp tt µ + tt Σt .
2
From the expression of its mgf above, we know that a multivariate normal distribution
is completely determined by its mean µ and covariance matrix Σ.

Why don’t we define X via its pdf?

• Recall the definition of the covariance matrix for a random vector. Any covariance
matrix Σp×p should be symmetric and nonnegative definite, i.e.,

at Σa ≥ 0, for any a ∈ Rp .

The B matrix in Σ = BB t can be viewed as the square root of Σ, but there are many
such square roots. So it is possible to obtain X1 and X2 from two different transforma-
tions, but they end up having the same distribution (see the example in John’s notes).

Any symmetric nonnegative definite matrix has a spectral decomposition

Σ = Γt ΛΓ, Λ = diag(λi )pi=1 ,

where λ1 ≥ λ2 · · · ≥ λp ≥ 0, and Γn×n is a orthonormal matrix, i.e., ΓΓt = In .

• If λp > 0, then Σ is positive definite, so |Σ| > 0 and Σ−1 = ΓΛ−1 Γt exists. Then the
pdf of Np (µ, Σ) is given by
 
1 1 t −1
f (x) = exp − (x − µ) Σ (x − µ) .
(2π)p/2 |Σ|1/2 2

3
Stat425 (F16) F. Liang

• Properties of the multivariate normal

– Affine transformations of multivariate normals are still normal:

X ∼ Nn (µ, Σ) =⇒ Am×n X + bm×1 ∼ Nm (Aµ + b, AΣAt ).

– Marginals of a normal are still normal.

– Conditionals of a normal are still normal.
 
X1 |X2 ∼ Nm µ1 + Σ12 Σ−1 22 (X 2 − µ2 ), Σ 11 − Σ Σ −1
Σ
12 22 21

– For multivariate normals, uncorrelated = independent.

Note: All the statements above assume that the joint distribution is normal. For
example,

X1 ∼ Norm, X2 ∼ Norm does NOT imply (X1 , X2 ) ∼ Norm.

So if X1 and X2 are marginally normally distributed with correlation 0, we cannot

conclude that X1 and X2 are independent.

4
Stat425 (F16) F. Liang

Distributions Related to Normal

• If Z ∼ Nn (0, In ), then kZk2 ∼ χ2n .

If Z ∼ Nn (0, σ 2 In ), then kZk2 /σ 2 ∼ χ2n .

If X ∼ Nn (µ, Σ) and Σ−1 exists, then (X − µ)t Σ−1 (X − µ) ∼ χ2n .

If X ∼ Nn (0, H) where H is a projection matrix (i.e., H is symmetric and idempotent),

then Xt HX ∼ χ2m with m = trace(H). Examples of H:
   1 1 
1 0 0 2 2 0
 0 1 0 ,  1 1 0 .
2 2
0 0 0 0 0 0

• Z ∼ N (0, 1) and W ∼ χ2n are independent, then

Z
√ ∼ Tn (student t-dist).
W

W1 ∼ χ2n , W2 ∼ χ2m and they are independent, then

W1
∼ Fn,m .
W2
Chi-square and Student t-dist have one df (degree of freedom) and F-dist has two dfs.

• X1 , . . . , Xn ∼ N(µ, σ 2 ), then X̄ and (X1 − X̄, . . . , Xn − X̄) are independent,

n
σ2 X
X̄ ∼ N(µ, ), (Xi − X̄)2 ∼ χ2n−1 ,
n
i=1

therefore
X̄ − µ
T = √ ∼ Tn−1 ,
S/ n
where S 2 = − X̄)2 /(n − 1) is the sample variance.
P
i (Xi

5
Stat425 (F16) F. Liang

Some Basic Concepts of Statistical Inference

Suppose we have a rv X that has a pdf/pmf denoted by f (x; θ) or p(x; θ), where θ is
called the parameter, e.g., p in Bern(p) and (θ, σ 2 ) in N(θ, σ 2 ).

Previously, we focus on problems where the value of θ is given, and we calculate various
quantities related to the distribution, e.g., the mean, the variance, and various probabilities.

Now we focus on problems where θ is unknown and we try to estimate various (unknown)
quantities related to this distribution, after observing a random sample (X1 , . . . , Xn ) from
this distribution.

Some jargons:

• Parameter θ

• Random sample: (X1 , . . . , Xn ) iid.

• Statistic T = T (X1 , . . . , Xn ): a function of the sample, which is also random.

• Estimator θ̂ = θ̂(X1 , . . . , Xn ) of θ is a function of the sample, i.e., a statistic. Given

an observed sample (X1 = x1 , . . . , Xn = xn ), the value of θ̂(x1 , . . . , xn ) is called an
estimate of θ. So, an estimator is a random variable, while an estimate is a real
number.

• Hypothesis testing: decide between the null hypothesis H0 : θ ≥ θ0 and the alternative
hypothesis Ha : θ < θ0 where θ0 denotes a fixed value for the parameter θ.

• Prediction

The Maximum Likelihood Estimator

• MLE: the estimator or estimators1 that maximize the Likelihood function
n
Y
L(θ; x) = f (x1 , . . . , xn ; θ) = f (xi ; θ).
i=1

• How to derive MLE? Write out the log likelihood function

n
hY i Xn
`(θ) = log f (xi ; θ) = log f (xi ; θ).
i=1 i=1

Find the maximum of `(θ): θ̂ = arg maxθ `(θ)

– Solve `0 (θ) = 0 (if no constraints);

– Otherwise, need to check whether the boundary points are the maximum.

1
MLE may not be unique.

6
Stat425 (F16) F. Liang

• Usually the likelihood function (i.e., the joint pdf/pmf) can be factorized into two
parts  
f (x1 , . . . , xn ; θ) = K1 s(x1 , . . . , xn ); θ · K2 (x1 , . . . , xn ),
where K2 (·) does not depend on θ, and s(x1 , . . . , xn ) is a summary statistic (e.g., sam-
ple mean/variance) of the data. Note that the conditional distribution of (X1 , . . . , Xn )
given S (if it exists) doesn’t depend on θ. In other words, the statistic S(X1 , . . . , Xn )
has exhausted all the relevant information about the unknown parameter θ contained
in this random sample, and this is why we call S(X1 , . . . , Xn ) a sufficient statistic for θ.

• Invariance property of MLE. Let X1 , . . . , Xn be a random sample with the pdf f (x; θ).
Let η = g(θ) be a parameter of interest. Suppose θ̂ is the mle of θ. Then g(θ̂) is the
mle of g(θ).

Bias, Variance, and MSE

• An estimator is called unbiased if E(θ̂) = θ.

• For an estimator θ̂ of θ, define the Mean Squared Error of θ̂ by

MSE(θ̂) = E(θ̂ − θ)2 = E[θ̂ − E(θ̂)]2 + Var(θ̂) = Bias2 + Var

Specially, if θ̂ is unbiased, then MSE(θ̂) = Var(θ̂).

If θ is multi-dimensional, then MSE is defined as Ekθ̂−θk2 = E|θ̂−E(θ̂)k2 +tr Cov(θ̂) ,
 

where the 2nd term is the trace of the covariance matrix of θ̂.

In Stat410/510, we have learned various methods for deriving estimators (MLE, method
of moment, Bayes, etc), and known how to compare estimators by MSE. A natural question:
can we find the best estimator(s), i.e., the estimator(s) with the smallest MSE? The answer
is NO.

Suppose we want to estimate θ, given X1 , . . . , Xn ∼ N(θ, 1). Consider the following two
estimators:
θ̂1 = X̄, θ̂2 = 0.
What does the MSE(θ), as a function of θ, look like for each of the two estimators? The
class of all possible estimators is too large, so it is impossible to find one whose MSE is
better than all other estimators for any value of θ. However, it is possible to find the best
estimator among a subclass of estimators, e.g., among all unbiased estimators. The best
unbiased estimators, i.e., the one with the smallest variance, is often referred to as the
UMVUE (uniform minimal variance unbiased estimator ). Later we will learn how to find
the UMVUE for linear regression models.

7
Stat425 (F16) F. Liang

Hypothesis Testing
• Given data X = (X1 , . . . , Xn ) ∼ P , we want to test

(null) H0 : P ∈ P0 , vs. (alternative) HA : P ∈ PA ,

or equivalently if all distributions have parameters,

H0 : θ ∈ Θ0 , vs. HA : P ∈ ΘA .

H0 : θ ≥ θ0 vs. Ha : θ < θ0 Left-tailed

H0 : θ ≤ θ0 vs. Ha : θ > θ0 Right-tailed
H0 : θ = θ0 vs. 6 θ0
Ha : θ = Two-tailed (or two-sided)

Our decision is usually made based on a test statistic δ(X): if δ(X) is in some region
(aka, the Rejection Region), reject H0 , otherwise do not reject.

• Two types of errors

H0 True H0 False
Do NOT Reject H0 ^¨ Type II Error

Reject H0 Type I Error ^

¨

• The power of a test δ is defined to be the probability of rejection when the true
parameter is θ, namely,

β(θ) = Pθ (δ(X) ∈ Rejection Region).

The significant level of a test (usually denoted by α) is defined to be maxθ∈Θ0 β(θ);

for two-tailed test, the significant level is β(θ0 ).
We usually fix the level of a test, say 5% test or 1% test, and then decide the Rejection
Region, i.e., the Rejection Region depends on α.

• The p-value (observed level of significance ) is the probability, computed assuming that
H0 is true, of obtaining a value of the test statistic as extreme as, or more extreme
than, the observed value.
Use p-value to perform a level α test: If p-value < α, reject; otherwise, do not reject
H0 .

• Connection between CIs and Hypothesis Tests. Suppose (L(X), U (X)) is a 100(1 −
α)% CI for θ. Consider

H0 : θ = θ0 , vs HA : θ 6= θ0 . (1)

Define a test: reject H0 if θ0 ∈ (L(X), U (X)). Then this is a test with significant level
α, since


Pθ0 ( Reject H0 ) = Pθ0 (L(X), U (X)) does not cover θ0 = α.

8
Stat425 (F16) F. Liang

Similarly we can invert tests to obtain CIs. Let δ(X) to denote the test statistic and
RRθ0 denotes the rejection region for level α test with null H0 : θ = θ0 . Given data
X, this set
B = {θ0 : δ(X) 6= RRθ0 } (2)
is a 100(1 − α)% CI2 for θ, since

Pθ (θ ∈ B) = P(Not Reject H0 | H0 is true) = 1 − α.

Here is the interpretation of this 95% CI (2): it contains θ values for which the null
would not be rejected given data X.

In most tests we face in Stat425, we are testing H0 : θ = θ0 and Ha : θ 6= θ0 , and

we usually have an unbiased estimator of θ, denoted by θ̂, which has standard error
sd(θ̂). The test statistic takes the following form

θ̂ − θ0
∼ Tν , under H0 .
sd(θ̂)

So for a level α test,  

 θ̂ − θ 
0
Reject H0 , if   > t(α/2) ,

ν
 sd(θ̂) 
(α/2)
where tν is the (1 − α/2) quantile of Tν . The (1 − α) CI for θ is

θ̂ ± t(α/2)
ν sd(θ̂).

It is easy to check that, given θ̂ (i.e., given a data set), for any θ0 in the (1 − α) CI,
we cannot reject the hypothesis H0 : θ = θ0 .

2
This set, if not an interval, is the credible region.