Statistical Inference Assignment

1. Find the log-likelihood of a sequence of i.i.d. Gaussian random variables X1 , . . . , XN with

mean µ and variance σ 2 .

2. Find the log-likelihood of a sequence of i.i.d. Bernoulli random variables X1 , . . . , XN with

parameter θ.

3. Find the ML estimate for a set of i.i.d. Bernoulli random variables {X1 , . . . , XN } with
Xn ∼ Bernoulli(θ) for n = 1, . . . , N .

4. Suppose that Xn is a Gaussian random variable. Assume that σ = 1 is known but the
mean θ is unknown. Find the ML estimate of the mean.

5. Suppose that we are given a set of i.i.d. Gaussian random variables X1 , . . . , XN , where
both the mean µ and the variance σ 2 are unknown. Let θ = [µ, σ 2 ]T be the parameter.
Find the ML estimate of θ.

6. Suppose that we are given a set of i.i.d. d-dimensional Gaussian random vectors X1 , . . . , XN
such that
Xn ∼ Gaussian(µ, Σ).
We assume that Σ is fixed and known, but µ is unknown. Find the ML estimate of µ.

7. Let X1 , . . . , XN be i.i.d. Gaussian random variables with an unknown mean θ. It has

been shown that the ML estimator is
N
1 X
θ̂ML = Xn .
N n=1

Is the ML estimator θ̂ML unbiased?

8. Let X1 , . . . , XN be i.i.d. random variables with a PDF fXn |Θ (xn |θ) for all n, and Θ be a
random parameter with PDF fΘ (θ):

(xn − θ)2
 
1
fXn |Θ (xn |θ) = √ exp − ,
2πσ 2 2σ 2
(θ − µ0 )2
 
1
fΘ (θ) = p exp − .
2πσ02 2σ02

Find the MAP estimate.

9. Let Yn = θ + Wn be the output of a noisy channel where the input is a scalar θ and
Wn ∼ N (0, 1) is an i.i.d. Gaussian noise. Suppose that we have observed y1 , . . . , yN .

(a) Express the PDF of Yn in terms of θ and yn . Find the joint PDF of Y1 , . . . , YN .
(b) Find the maximum likelihood estimate θ̂ML . Express your answer in terms of
y1 , . . . , y N .
(c) Find E[θ̂ML ].

1
10. The Kullback-Leibler divergence between two distributions p1 (x) and p2 (x) is defined as
Z
p1 (x)
KL(p1 ∥p2 ) = p1 (x) log dx.
p2 (x)

Suppose we approximate p1 using a distribution p2 . Let us choose p2 = Gaussian(µ, Σ).

Show that µ and Σ, which minimize the KL divergence, are such that

µ = Ex∼p1 (x) [x] and

Σ = Ex∼p1 (x) [(x − µ)(x − µ)T ].

11. Let X1 , . . . , XN be a sequence of i.i.d. Gaussian random variables with unknown mean
θ1 and variance θ2 . Suppose that we have observations x1 , . . . , xN .

(a) Express the PDF of Xn in terms of xn , θ1 and θ2 . Find the joint PDF of X1 , . . . , XN .
(b) Find the maximum likelihood estimates of θ1 and θ2 .

12. Consider a deterministic vector s ∈ Rd and random vectors

fY|Θ (y|θ) = Gaussian(sθ, Σ),

fΘ (θ) = Gaussian(µ, σ 2 ).

(a) Show that the posterior distribution is given by

fΘ|Y (θ|y) = Gaussian(m, q 2 ),

where

d2 = sT Σ−1 s,
 −1 
1 µ
m= d + 22
sT Σ−1 y + 2 ,
σ σ
1
q2 = 2 .
d + σ12

(b) Show that the MMSE estimate θ̂MMSE (y) is given by

σ 2 sT Σ−1 y + µ
θ̂MMSE (Y) = .
σ 2 d2 + 1

13. Let ( (
θe−θx , x ≥ 0, αe−αθ , θ ≥ 0,
fX|Θ (x|θ) = and fΘ (θ) =
0, x < 0, 0, θ < 0.
Find the ML, MAP, and MMSE estimates for a single observation X = x.

14. Let X1 , . . . , XN be a sequence of i.i.d. Bernoulli random variables with P [Xn = 1] = θ.

Suppose that we have observed x1 , . . . , xN .

(a) Show that the PMF of Xn is pXn (xn |θ) = θxn (1 − θ)1−xn . Hence, find the joint PMF

pX1 ,...,XN (x1 , . . . , xN ).

2
 (b) Find the maximum likelihood estimate θ̂, i.e.,

θ̂ML = argmax log pX1 ,...,XN (x1 , . . . , xN ).

θ

Express your answer in terms of x1 , . . . , xN .

15. Let Yn = θ + Wn be the output of a noisy channel where the input is a scalar θ and
Wn ∼ N (0, 1) is an i.i.d. Gaussian noise. Suppose that we have observed y1 , . . . , yN .

(a) Express the PDF of Yn in terms of θ and yn . Hence, find the joint PDF of Y1 , . . . , YN .
(b) Find the maximum likelihood estimate θ̂ML . Express your answer in terms of
y1 , . . . , y N .