Notes on intermediate statistics

Aodong Li

January 4 - 20, 2018

*This document only serves as notes on intermediate statistics 36-705 CMU.*

1 Probability Review

Random variable is a function that maps the outcomes in sample space into R.

Conditional expectation is a random variable.

An important theorem is iterated expectations

E[E[f (X)|Y ]] = E[f (X)].

The moment generating function E[etX ] completely characterizes a random variable. It

means that if E[etX ] = E[etY ], then X and Y have the same distribution.

To generate a moment,
dk E[etX ]
E[X k ] = |t=0 .
dtk

Exercise 3.16. Show that E[r(X)s(Y )|X] = r(X)E[s(Y )|X] and that E[r(X)|X] = r(X).
h(x) = E[r(X)s(Y )|X = x]
= E[r(x)s(Y )|X = x]
= r(x)E[s(Y )|X = x]
h(X) = r(X)E[s(Y )|X]

f (x) = E[r(X)|X = x]
= E[r(x)|X = x]
= r(x)
f (X) = r(X)

1
Exercise 3.18. Show that if E[X|Y = y] = c for some constant c, then X and Y are
uncorrelated.

Cov(XY ) = E[XY ] − E[X]E[Y ]

= E[E[XY |Y ]] − E[E[X|Y ]]E[Y ] (by iterated expectations)
= E[Y E[X|Y ]] − E[E[X|Y ]]E[Y ] (by exercise 3.16)
= cE[Y ] − cE[Y ]
=0

Exercise Moments generating function. Show that if X1 , ..., Xn ∼ N (µ, σ 2 ), then X̄n ∼
N (µ, σ 2 /n).
n
Y
E[etX̄n ] = E[ etXi /n ]
i=1
n
Y
= E[etXi /n ]
i=1
n
Y µt σ 2 t2
= exp( + )
i=1
n 2n2
σ 2 t2
= exp(µt + )
2n
This means that X̄n ∼ N (µ, σ 2 /n).

2 Inequalities

The Gaussian Tail Inequality, Markov’s Inequality, and Chebyshev’s Inequality show similar
properties of distributions–to what probability that the random variable lies in a far-away
area P (|X| > t).

The Gaussian Tail Inequality is much more tighter since it introduces much more infor-
mation about the details of the distribution.

We can use more moments to better bound the probability, which comes to the Chernoff ’s
method. Because the moments generating function encodes all the moments of a random
variable (Taylor expansion shows all the moments), it’s more appropriate to use E[etX ]

P (X > ) = P (etX > et ) ≤ inf e−t E[etX ]

t≥0

where t > 0.

2
 The derivatives show an very important trick called variational trick by introducing
more variables, here it’s t. This renders much more flexibility, for example, minimizing the
right size with respect to t gives a much tighter bound.

We will not be able to compute the moments generating function for all different distri-
butions, so we might as well bound it.

Note that a ≤ X ≤ b implies that all of X’s moments are bounded! The important result
is
t2 (b−a)2
E[etX ] ≤ etµ e 8

where µ = E[X].

Hoeffding’s Inequality gives a very much tight bound on sum of independent vari-
ables and shows that the convergence is very quick. Hoeffding’s Inequality also defines the
confidence interval.

A function of a set of independent variables also shows such properties if the function is
wiggling within some constant. That is demonstrated by The Bounded Difference Inequality
or McDiarmid’s inequality.

log function in KL distance is a concave function from which we can utilize the Jensen’s
inequality.

The cdf of the maximum of a set of random variables can be calculated by requiring every
random variable to be less than m. But this is often not possible in practice.

The expectation of the maximum of a set of random variables grows at log n rate by
Theorem 16.

an = oP (1) means that an eventually converges to zero as n → ∞. an = OP (1) means

an is eventually bounded, i.e., |an | ≤ C for some C > 0.

For probabilistic versions, if Yn = oP (1), then for every  > 0,

P (|Yn | > ) → 0.
It says that as n gets large, the probability mass concentrates around zero.

if Yn = OP (1), then for every  > 0, there exists a C > 0 such that
P (|Yn | > C) ≤ .
It says that as n gets large, the probability mass is bounded. It can also be explained as that
first choose an arbitrary , we can find a C satisfying the inequality. Consider the shifted
probability mass sequence Yn ∼ N (n, 1). It is bounded and Yn = OP (1).
√ √
p̂n − p = OP (1/ n) by Hoeffding’s Inequality. The standard deviation ∼ 1/ n.

OP (·) gives us more information.

3
Exercise OP (1) and oP (1). Suppose xn = OP (1), yn = oP (1), and zn = xn yn , all of these
can be proved.

OP (1)oP (1) = oP (1)

OP (1)OP (1) = OP (1)
oP (1) + OP (1) = OP (1)
OP (an )oP (bn ) = oP (an bn )
OP (an )OP (bn ) = OP (an bn )

3 Uniform bound and VC dimension

Empirical cdf can be seen as a Bernoulli variable. So we can use Hoeffding’s Inequality.
2
P (|Fn (t) − F (t)| > ) ≤ 2e−2n for every t ; P (supt |Fn (t) − F (t)| > ) ≤ something.
This is an example of the difference between pointwise convergence and uniform convergence.
It relates to the issue of overfitting1 . Check infimum speed over all points to determine
whether it is uniform convergence.
1
[Link] In general,
pointwise convergence is looser than uniform convergence. Pointwise convergence says at every point the
sequence of functions has its own speed of convergence (that can be very fast at some points and very very
very very slow at others), but uniform convergence says there is an overall speed of convergence.

4
 P (subA∈A |Pn (A) − P (A)| > ) = P (∪N
i=1 |Pn (Ai ) − P (Ai )| > )

The classes A can have many different forms.

The situation is separated from finite classes and infinite classes.

4 Convergence

The convergence is all about a sequence of statistics. The sequence does not need to be iid,
but the underlying sequence is iid. They are not independent.

In probabilistic realm, limn→∞ Xn = c is an event, so we have to bound it using proba-

bility.

The convergence in probability says Xn is concentrating around X but not Xn = X

numerically. In particular, if Xn → c, then the distribution of Xn concentrates around c and

5
is sharp.
Xn − c = oP (1)

Convergence in probability is the central role.

Convergence in distribution holds at all t where F (t) is continuous. It doesn’t matter

where F (t) is not continuous.

Convergence in distribution is another central role.

Sometimes it is easier to prove the convergence in quadratic mean, which implies the
convergence in probability.

If a random variable converges to a point mass in distribution, then the convergence in

probability also holds.

Point mass distribution δc (t) 0 if t < c and 1 if t ≥ c.

√ √
Let Xn = nI(0 < U < n1 ), Xn can only take two values 0 or n.

Convergence in quadratic mean is stronger in that it requires the probability mass doesn’t
go far away while convergence in probability implies probability concentrates around some
point (and doesn’t care if the probability goes far away). More precisely, convergence in
probability does not consider the moments. Any statement that involves moments is stronger.

Let Xn = −X for n = 1, 2, 3, ..., this is a well defined sequence but highly correlated and
it’s a sequence of a single value. Xn is defined over a common sample space and remember
that random variable is a function that maps every outcome into R. Xn for all n are always
the same function, so whenever an outcome is revealed the mapped value is determined.
Make sure to distinguish the difference between Xn = X and Xn equals X in distribution.

Convergence in distribution can be used to calculate the probabilistic statements by

substituting a convergence distribution.

Exercise Convergence is preserved under transformations. If Xn →P X and Yn →P

Y , then Xn + Yn →P X + Y .
 
P (|Xn + Yn − X − Y | ≥ ) ≤ P (|Xn − X| ≥ ∪ |Yn − Y | ≥ )
2 2
 
≤ P (|Xn − X| ≥ + P (|Yn − Y | ≥ (by union bound)
2 2
→ 0.

Not all random variables have a moment generating function.

Moments generating function is smooth.

6
 CLT can be proved by moments generating function. It turns out that the mgf of X̄n
has the form of Gaussian mgf.

Berry-Esseen√Theorem tells how close the distribution of X̄ is to the Normal distribution

and why it has n term.

If we replace σ with sample standard deviation, CLT still holds.

√
n(X̄n − µ)
∼ N (0, 1)
Sn
This can be proved by Slutzky’s theorem and continuous mapping theorem through
√
n(X̄n − µ) σ
σ Sn

To measure the distance between two distributions, we can use

sup |X(z) − Y (z)|

z

For multivariate central limit theorem, the vectors are iid instead of elements in the
vector. Instead, the elements can be highly correlated, like, height and weight of a person.

The distribution of g(X̄) cannot be directly cast into any distribution scaled by g(·)
because there is a scaling issue caused by the sum of one constraint.

Exercise See why the Delta Method holds. By Taylor’s theorem

g(x̄) ≈ g(µ) + (x̄ − µ)g 0 (µ).

√ √
n(g(x̄) − g(µ)) ≈ n(x̄ − µ)g 0 (µ)
√
n(X̄ − µ) → N (0, σ 2 )
√
n(x̄ − µ)g 0 (µ) → N (0, g 0 (µ)2 σ 2 )

5 Sufficiency

Statistics is kind of the reverse of probability. It is about given data and infer the distribution.

Parametric model can be parameterized by a finite number of parameters.

Parametric models is simple but can de dangerous since we assume a distribution that
might not be the true case.

7
 Parametric model is a good start because it can be seen as building blocks for complicated
nonparametric models.

X̄ − µ is not a statistic because it is not a function of data, instead, it is a function of

unknown parameters.

f (X; θ) is a family of density indexed by parameter θ (Note the semicolon).

Sufficiency is only useful for parametric model.

P (X1 = x1 , ..., Xn = xn , T = t) = P (X1 = x1 , ..., Xn = xn ) because the intersection of

the set {X n |T = t} and the set {X n |X1 = x1 , ..., Xn = xn } is just the latter, which represents
a point.

Statistics are equivalent if they produce the same partitions of the data.

If we can derive a conclusion from other statistics, it can be seen that other statistics
contain more information or redundant information.

Minimal sufficient statistic generates the coarsest sufficient partition.

If we have a sufficient statistic, we can compute the likelihood function from it.

6 The likelihood function

The likelihood function serves the purpose of generating estimators, Bayesian inference, and
sufficiency.

Exercise Proof of Theorem 4. if L(θ|xn ) ∝ L(θ|y n ), then the likelihood function has the
same shape no matter what the parameter is. This means xn and y n are in the same
partition such that T (xn ) and T (y n ) should be equal. Intuitively, this introduces a
minimal sufficient partition. This is indeed the case.

Technically,

p(xn ; θ)
R(xn , y n ; θ) =
p(y n ; θ)
L(θ|xn )
=
L(θ|y n )
=c

R(xn , y n ; θ) does not depend on θ. By Theorem 10 in lecture note 5, this is indeed the
case.

8
 Then we can say that the likelihood function contains the information of minimal suffi-
ciency.

7 Parametric point estimation

In frequentist view, the parameter is fixed, unknown constant but the estimator is a random
variable, a function of sampled data.

Estimating ”mixture of Gaussian” using the method of moments is tractable than MLE.

Minimax theory is perhaps the most important field of defining the what an optimal
estimator is and it provides a formal way.

Consistency of the estimator should be a minimal requirement!

Method of moments: Equate the sample moments with the theoretical moments by the
law of large number.

The estimator of k for Binomial distribution might be a bad estimator since the denom-
inator can be negative.

The log-likelihood changes the shape of the function but the maximum is the same place.

MLE under certain conditions is the optimal.

Sometimes MLE need to be found by numerical methods.

Find the profile likelihood supξ L(η, ξ) implies that finding the maximum along η. The
result is still a set. Maximizing the set gives η̂.

MLE has a very important property called equivariant. This might be used to construct
concave function. Even though the parameter is transformed, the maximum of the likelihood
function does not change.

Equivariance for MLE is a very good property that does not share with other estimators.

Unbiased estimator does not have the equivariant property.

Bayes estimator is not Bayes inference.

Every different prior distribution gives a different Bayes estimator.

We can also take the moments of the posterior distribution.

Mean square error is the overall property of the estimators.

9
 Nowadays unbiased estimators are not taken so important.

Convergence of MSE to zero with speed of O( n1 ) is the property of some parametric

models. But nonparametric models always have a slower convergence speed.

Computing MSE is a useful thing but it does not quite solve the problem since it always
include the unknown parameter.

8 Minimax theory

Classification usually uses zero-one loss.

MSE is a special case of a more general concept.

The risk of an estimator is a function of parameter θ.

R(θ, θ̂) = g(θ)

Maximum risk is the worst case of the risk.

First we can find the minimax risk and then find an estimator equal to the minimax risk.
This estimator is a minimax estimator.

Instead of the supremum of the risk, we can also take the expected value of it.
P
X+α
For the estimator α+β+n
, every α and β gives rise to a valid Bayes estimator.

Different from Maximum risk, which uses the maximum of the risk, Bayes risk takes the
weighted average of the risk. But Bayes risk is dependent on the choice of prior distribution.
Z
Bπ (θ̂) = R(θ, θ̂)π(θ)dθ

and Bayes estimator minimizes Bayes risk.

Unlike the minimax risk, Bayes risk is easier to calculate and if we choose π(·)
carefully, Bayes estimator is minimax estimator. So the process is that first we
calculate a Bayes estimator and then choose a prior distribution that makes it a
minimax estimator.

Minimax risk Rn usually reaches zero as n approaches ∞, but we consider more on at

what rate it goes to zero. It can be seen that for parametric learning the speed that Rn
reaches zero is faster than nonparametric learning.

The main focus is that choosing the Bayes estimator as a root for minimax estimator
with a proper prior.

10
 Posterior risk takes the expected value with respect to fθ|xn while the risk takes the
expected value with respect to fxn |θ .

Bayes risk can also be written as

Z
Bπ (θ̂) = r(θ̂|xn )π(xn )dxn

which is nonnegative. This gives a way to compute Bayes estimator.

For theoretical reason, we always use the formal definition of Bπ (θ̂) but for computation
aspect, we always use the posterior form of Bπ (θ̂).

A Bayes estimator minimizes the posterior risk. It’s kind of like if I want to minimize
the sum, minimizing each element of the sum minimizes the whole sum.

The mean of the posterior E[θ|X = xn ] is the Bayes estimator for L2 loss. This is still a
function of the prior.

Theorem The bound for the minimax risk. Suppose Rn = inf θ̃ supθ R(θ, θ̃) is the min-
imax risk.
Bπ (θ̂) ≤ Rn ≤ sup R(θ, θ̃0 )
θ

where θ̂ is an arbitrary Bayes estimator for any given prior π(θ) and θ̃0 is an arbitrary
estimator.

Proof. The right-hand side of the inequality is obvious, since the minimum of a function is
always less than or equal to values at any other point.

The left-hand side of the inequality can be proved.

Z
inf sup R(θ, θ̃) ≥ inf R(θ, θ̃)π(θ)dθ
θ̃ θ θ̃

= Bπ (θ̂)

since maximum of a set is always larger than a weighted average of the set.

MLE is approximately minimax for parametric models if the number of pa-

rameters is fixed. MLE is approaching minimax as n is increasing. For now, we
can take that MLE is minimax.

Bayes estimator with a constant Bayes risk function is minimax.

Theorem X̄n is minimax under squared error loss. Suppose X1 , ..., Xn ∼ N (θ, 1) and
θ ∼ N (0, c2 ), then θ = X̄n is minimax under squared error loss.

11
Proof. Take the estimator θ̂0 = X̄n ,
1
sup R(θ, θ̂0 ) = V ar[X̄n ] =
θ n
then Rn (θ̂) ≤ supθ R(θ, θ̂0 ) = n1 .

Then take the Bayes estimator under squared error loss, which is the posterior mean
nc2 x̄
θ̂1 = E[θ|X = xn ] =
1 + nc2

nc4
V ar[θ̂1 ] =
(1 + nc2 )2
bias(θ̂1 ) = Eθ [θ̂1 ] − θ
θ
=
1 + nc2
R(θ, θ̂1 ) = E[(θ − θ̂1 )2 ]
= V ar[θ̂1 ] + bias(θ̂1 )2
θ2 + nc4
=
(1 + nc2 )2
Z
Bπ (θ̂1 ) = R(θ, θ̂1 )π(θ)dθ
c2 (1 + nc2 )
=
(1 + nc2 )2
c2
=
1 + nc2
1
Rn (θ̂) ≥ Bπ (θ̂1 ) →
n
where c → ∞ because it holds for every c.

So Rn = n1 . We can find an estimator θ whose minimax risk is n1 .

1
sup R(θ, X̄n ) =
θ n
thus by the definition of the minimax estimator, X̄n is minimax under squared error loss.

9 Asymptotic theory

Probably the most important distance is Kullback-Leibler distance

Z
p(x)
KL(P ||Q) = p(x) log dx
q(x)

12
and Hellinger distance sZ
p p
h(P, Q) = ( p(x) − q(x))2 dx

To make MLE consistent, the premise is the model is parametric. And (1) the di-
mension of the parameter space does not change with n; (2) p(x; θ) is a smooth
function of θ.

MLE is very naturally related to KL distance.

The likelihood is bigger at the point of the true parameter than other points.

L(θ̂)
P( > 1) → 1
L(θ)

Thus Maximum Likelihood Estimator is a consistent estimator, which means it always gives
the true estimate under regularity conditions.

MLE is effective for situations like the number of data is large and the number of param-
eter is small.

MLE converges to Normal distribution. In fact, other estimators also converge to Normal
distribution but with a larger variance. That is why we call the mle is optimal or efficient.

Score function and Fisher information comes up very often in maximum likelihood theory.

Take the score function, the derivative of log likelihood, Sn (θ) = 0 usually gives the mle.

Score function is a random function. It is a function of θ but also depends on the data.
So given any θ, Sn (·) is a random variable of X and we can take the mean or variance on it.

1
V ar(θ̂) ≈
In (θ)
Why In (θ) is called information is that if we have a lot of information, In (θ) is large, the
variance is small and the estimator is precise.

For U nif orm(0, θ), its support of p changes with θ.

Since the Fisher information In (θ) = nI(θ), as we get more and more data, the informa-
tion adds up.

If without regularity conditions, such as U nif orm(0, θ), the Score function and Fisher
information are useless. They are only dealing with well-behaved parametric models.

We usually use asymptotic normality to compute the confidence interval.

13
 An counterexample: X1 , ..., Xn ∼ U nif orm(0, θ), then mle is θ̂. It turns out that
√
n(θ − θ̂) → Exponential

The estimated standard error for the mle estimator

s
1
se
ˆ =
In (θ̂)

is probably the most important approximation.

Relative efficiency is used to compare estimators, because some estimators are easier to
compute than MLE.

As we can see from the Example 17, different estimators might be valid and mle is always
the more efficient. This in only true when the model is correct. But if the true model is not
what you assume, mle may gives the biased estimate while estimate from LLN gives unbiased
result. Moral: there is a trade-off between precise and efficiency. It depends on
the extent to which you trust your model.

MLE is optimal only in the sense that the model is true.

The real stuff is the nonparametric statistics.

All the above facts lie in that the parametric model is correct. But the parametric model
is not correct.

We can trade efficiency for robustness.

10 Hypothesis testing

The question is whether there is sufficient evidence to reject H0 .

In general, false positive error are worse and we do not want to make false positive error.

First and foremost, we need to choose an error rate for Type I error based on the cases
that we are concerned. For example, the null hypothesis is the mountain spring is harmful
for students. We definitely want to reject it not easily. That corresponds to a low Type I
error.

1. Choose a proper test statistc and 2. choose a proper rejection region.

Neyman-Pearson test is not usually used in practice but it is very important for theoretical
reason.

14
 α is the amount of Type I error that we can tolerate.

Null hypothesis is more specific. In Neyman-Pearson test, the alternative hypothesis is

also specific, like two sides coin. That is counter to practical situations.

For the Neyman-Pearson test, it says if we have another size α test with power function
β
βN P (θ1 ) > β(θ1 )

Wald test, likelihood ratio test, and permutation test is more common in practice.

For the Wald Test, the goal is to test a scalar and note that θ̂ is the mle estimator under
null hypothesis. There are two different ways to compute se. You can use the estimated
standard error of θ̂, or on the other hand, you can use the standard error under H0 , because
θ̂ converges to θ asymptotically.

If the true value is θ, then we can rewrite Tn

θ̂ − θ0 p
= nI(θ)(θ̂ − θ0 )
se
ˆ p p
= nI(θ)(θ̂ − θ) + nI(θ)(θ − θ0 )
The first term is converging to N (0, 1) and the second term is getting larger and larger such
that β(θ) → 1 for θ ∈ H1 .

For the likelihood ratio test, it can be used to test vectors. And unlike Neyman-Pearson
test, it can be used for composite null hypothesis and composite alternative hypothesis.

Finding the maximum of the likelihood function can use numerical methods by sampling
all values of θ.

Rearrange the reject region for the likelihood ratio test λ(x1 , ..., xn ) < c into |Tn | > k
where √
n(X̄ − θ0 )
Tn =
S
This is kind of the Wald test. Under H0 , Tn has a t-distribution with n-1 degrees of freedom.
The Student’s t-distribution is more accurate when n is small. But it is useless we won’t
conduct such computation when n is small.

p value basically can be seen as cdf. And if H0 holds, p conforms to Uniform distribution.

Permutation test is distribution free. So sometimes working out the distribution under
null hypothesis is very hard and we can use permutation instead.

Any continuous cdf conforms to uniform distribution [0,1]. This can be demonstrated by
the inverse transform theorem Y = FX−1 (U ) = FY (y).
FFY (y) = FU (y)

15
11 Confidence sets

Remember, the parameter is not random, instead, the procedure of generating the sets is
random.

What we usually do is to construct a test first, and then convert the hypothesis testing
into a confidence set.

Theorem Confidence interval for MLE.

√
n(θ̂ − θ)
∼ N (0, 1)
σ̂
√
The confidence interval is Cn = θ̂ ± σ̂zα/2 n such that
P (θ ∈ Cn ) → 1 − α

The confidence intervals constructed by probability inequalities are quite conservative

and so they’re not used much since the interval is quite wide and uninformative.

The Wald Interval under the asymptotic approximate is the most commonly used interval.

When α = .05, zα/2 ≈ 1.96 ≈ 2, if θ̂n is the MLE, then 95% confidence interval is

θ̂n ± 2se

Confidence intervals are more informative. Once we have a confidence interval, we can
test a hypothesis.

The Wald test interval is symmetric but the likelihood test interval is not. In some sense,
the likelihood test interval is more accurate than the Wald interval.

12 Nonparametric Inference
1
For histogram density estimator, the convergence speed is O( n−2/3 ). For the kernel density
C1
estimator, the risk is n4/5 .

For high dimension situation, say d-dimension, the kernel density estimator is
1X 1 x − Xi
p̂(x) = d
K( )
n h h
and the risk function is
C2
R = C1 h4 +
nhd
16
the minimum risk is
C1
R= 4/(4+d)
n
So we can see that as the d gets larger. The estimator gets poor very quickly. This is a
glimpse of the curse of the dimension.

The empirical cdf can be seen as putting mass 1/n at each Xi

1X
Fn (x) = I(Xi ≤ x)
n

PCA is a statistical functional of distribution–calculate the covariance matrix and do the

eigendecomposition.

Plug-in estimator is a way to compute the statistical functional from empirical cdf.

To decide when a plug-in estimator is a good estimator is a very complicated subject. A

lot of time they are not good estimators.
R
Now to use the plug-in estimator to estimate the mean, we use xdFn (x).
But the empirical cdf Fn (x) is discrete, so the integral should be interpreted as
P
xp(x) where p(x) is the probability mass function. The key here is the empirical
cdf Fn (x) can be interpreted as putting mass 1/n at each data point, which is the
pmf. The final result is Z X 1
xdFn (x) = Xi = X̄
n

For most cases, the statistical functional θ̂n = T (Pn ), then

θ̂n − θ
→ N (0, 1)
se
ˆ
so we can use the nonparametric version of Wald test.

The plug-in estimator might be useful for the simple functional like the mean X̂ and the
2
variance Sn , but how about the complicated functional like the maximum eigenvalue and the
covariance matrix. That needs bootstrap.

The plug-in estimator for the mean and the variance happens to be the MLE estimator
for parametric model.

Plug-in estimator for computing the pdf from empirical cdf fails and that is a case when
plug-in estimator fails.

17
13 Bootstrap

If we know the true distribution P of data, then we can simulate the variance by the Law of
Large Number Theorem.
N
1 X
θi → E[θ]
N i=1
N
1 X 2
θ → E[θ2 ]
N i=1 i
N N N
1 X 2 1 X 2 1 X 2
(θi − θ̄) = θ −( θi )
N i=1 N i=1 i N i=1
→ E[θ2 ] − (E[θ])2
= V ar[θ]
where θi = g(Xi1 , Xi2 , ..., Xim ). In fact, θi means the ith m-number sample from the data.
In total, we sample N set of samples of size m. But we do not know the true distribution
P . Use the empirical distribution Pn ! That’s the bootstrap.

In all, we estimate Sn (P ) with Sn (Pn ). The simulation part is not difficult, the real
difficult part is replace P with Pn .

Drawing data with replacement ensures the data are iid.

√ √
For Op (1/ n), we √ cannot control. But for Op (1/ B), we can increase B to decrease the
error. So in all, Op (1/ n) dominates.

Estimate the cdf of the estimator using the bootstrap!

14 Bayesian inference

The prior distribution in Bayesian inference is different from the one used for deriving the
minimax estimator. Bayesian prior is used to express the subjective belief while the one used
in minimax theory might be uninformative and fails to capture the subjective belief.

For Bayesian inference, you don’t have to use Bayesian theorem. You can just look at
the data and come up with the posterior distribution.

In contrast, in Frequentist inference, you can also use Bayes theorem to find a frequentist
estimator like Minimax estimator, which is also a Bayes estimator.

Most of time, the integral in Bayesian inference is intractable. In practice, we resort to

Monte Carlo methods.

18
 As n gets larger and larger, the prior disappears.

Usually, ”non-informative priors” is not transformation invariant, but Jeffreys priors is.

If you come up with a 95% Bayesian confidence interval, it does not mean you can capture
the true parameter with 95% with repeated procedures. Instead, every time the parameter
might be very different.

In Bayesian inference, the consistency means the posterior distribution converges to or

concentrates to the true value.

19