Moment Inequalities
• Schwarz’s Inequality (sometimes called Cauchy-Schwarz)
(E[XY ])2 ≤ E[X 2]E[Y 2]
Proof. Suppose that E[X 2] > 0 and E[Y 2] > 0 Let
X Y
U=p and V = p
E[X 2] E[Y 2]
It can be shown that 2|U V | ≤ U 2 + V 2. Thus
2|E[U V ]| ≤ 2E[|U V |] ≤ E[U 2] + E[V 2] = 2
This gives
(E[U V ])2 ≤ (E[|U V |])2 ≤ 1
Moment Inequalities 1
implying
(E[XY ])2 (E[|XY |])2
≤
E[X ]E[Y ] E[X 2]E[Y 2]
2 2
2
One consequence of this inequality is that (Cov(X, Y ))2 ≤
Var(X)Var(Y ) or |Cov(X, Y )| ≤ σX σY . A consequence of this is
that |Corr(X, Y )| ≤ 1, a result discussed earlier.
Moment Inequalities 2
• Jensen’s Inequality
If g(·) is a convex function on the interval (a, b) and X is a RV taking
values in (a, b), then E[g(X)] ≥ g(E[X]).
Note that a function g(·) is convex on
the open interval I = (a, b) if Convex function
12
10
8
g(αu+(1−α)v) ≤ αg(u)+(1−α)g(v)
g(x)
6
4
2
for all u, v ∈ I and 0 ≤ α ≤ 1.
0
0 u 1 2 3 v 4
Moment Inequalities 3
Proof. Convexity means that Convex function
a supporting line exists at
12
each t ∈ (a, b). i.e. the
10
graph lies completely above
8
each tangent line.
g(x)
6
From the supporting line at
4
t = E[X] (with slope λ), we
2
have
0
0 1 X 2 3 4
g(x) ≥ g(E[X]) + λ(x − E[X])
E[g(X)] ≥ E[g(E[X]) + λ(X − E[X])
= g(E[X]) + λ(E[X] − E[X]) = g(E[X])
Moment Inequalities 4
A couple of examples where Jensen’s inequality can be used are the following
1. E[eX ] ≥ exp(E[X]).
For example, assume X ∼ N (µ, σ 2) and let Y = eX ∼ logN (µ, σ 2).
A consequence is that E[Y ] = E[eX ] ≥ eµ.
µ+0.5σ 2
In fact E[Y ] = e
Note going the other way, we get log(E[X]) ≥ E[log X] since − log x is
a convex function (log x is a concave function).
2
i.e. log eµ+0.5σ ≥ µ
Moment Inequalities 5
2. Arithmetic mean ≥ Geometric Mean ≥ Harmonic Mean
For any set of n positive numbers x1, x2, . . . , xn,
x1 + . . . + xn √ n
≥ n x1x2 . . . xn ≥ 1
n x1 + . . . + x1n
To justify the first inequality let X be a random variable taking values
x1, x2, . . . , xn each with probability n1 . Then Jensen’s says
µ ¶
x1 + . . . + xn log x1 + . . . + log xn
log ≥ = log(x1 . . . xn)1/n
n n
Then exponentiate both sides to get the first inequality.
The other inequalities can be derived similarly.
Moment Inequalities 6
• Lyapunov’s Inequality
If 0 < s < t
(E[|X|s])1/s ≤ (E[|X|t])1/t
A consequence of this is the relationship (for some integer p)
E[|X|] ≤ (E[|X|2])1/2 ≤ (E[|X|3])1/3 ≤ . . . ≤ (E[|X|p])1/p
which implies
|E[X]|q ≤ (E[|X|])q ≤ E[|X|q ] if 1 ≤ q ≤ p
Proof. Let r = st > 1. Let Y = |X|s and apply Jensen’s inequality to
g(y) = |y|r , giving (E[|Y |])r ≤ E[|Y |r ]. This implies that
(E[|X|s])t/s ≤ E[|X|t]
Taking the tth root of each side gives the result. 2
Moment Inequalities 7
Probability Inequalities
• Markov Inequality
Let X be a non-negative RV (i.e. P [X ≥ 0] = 1). Then for any a > 0,
E[X]
P [X ≥ a] ≤
a
Proof.
Markov Proof
10
X ≥ XI{X ≥ a} ≥ aI{X ≥ a} X
X I{X >= a}
8
a I{X >= a}
6
Therefore
g(x)
E[X] ≥ E[XI{X ≥ a}] = aP [X ≥ a]
2
0
2 0 2 a 4 6 8 10
x
Probability Inequalities 8
Note that there is an alternative version of this inequality that says if
[X r ] < ∞,
E[X r ]
P [X ≥ a] ≤
ar
• Chebyshev’s Inequality.
If E[X] = µ and Var(X) = σ 2 < ∞, then
σ2
P [|X − µ| ≥ k] ≤ 2
k
Note that this equality is sometimes written as the equivalent
1
P [|X − µ| ≥ kσ] ≤
k2
Probability Inequalities 9
Proof.
2
2 2 E[(X − µ) ]
P [|X−µ| ≥ a] = P [(X−µ) ≥ a ] ≤ by Markov’s Inequality
a2
Take a = k to get the first form of the result and a = kσ to get the
second form of the result. 2
Example: Suppose it is known that the number of widgets produced for
Guinness breweries in a factory during an hour is a RV with mean 500.
1. What can be said about the probability that an hour’s production will
exceed 1000?
Answer: By Markov’s inequality
E[X] 500
P [X ≥ 1000] ≤ = = 0.5
1000 1000
Probability Inequalities 10
2. If the variance of a hour’s production is known to be 100, then what
can be said about the probability that a hour’s production will be
between 450 and 550?
Answer: By Chebyshev’s inequality
Var(X) 100 1
P [|X − 500| ≥ 50] ≤ = 2= = 0.04
502 50 25
This implies that
1 24
P [|X − 500| < 50] ≥ 1 − = = 0.96
25 25
3. What can be said about the probability that the production will be
between 450 and 550 if X is normally distributed (N (500, 100))?
· ¸
450 − 500 550 − 500
P [450 ≤ X ≤ 550] = P ≤Z≤
10 10
= P [−5 ≤ Z ≤ 5] = Φ(5) − Φ(−5) = 0.9999994
Probability Inequalities 11
Note that these bounds are not particularly tight in most cases.
In fact they are what happens in a “worst case scenario”.
The following inequality also fits into this setting, where the bounds are
often loose.
• One-sided Chebyshev’s Inequality
If E[X] = µ and Var(X) = σ 2 < ∞, then for any a > 0,
σ2
P [X ≥ µ + a] ≤ 2
σ + a2
σ2
P [X ≤ µ − a] ≤ 2
σ + a2
Probability Inequalities 12
Proof. Without loss of generality, assume that µ = 0. Then for any b,
P [X ≥ a] = P [X + b ≥ a + b]
= P [(X + b)2 ≥ (a + b)2]
E[(X + b)2] E[X 2] + b2
≤ =
(a + b)2 (a + b)2
α + t2 Def
= = g(t)
(1 + t)2
where
E[X 2] σ 2 b
α= = 2; t=
a2 a a
To minimize g(t) (i.e. find the best b), set t = α, yielding
2 σ2
α+α a2 σ2
min g(t) = 2
= 2 =
(1 + α) σ
1 + a2 σ 2 + a2
Probability Inequalities 13
The other inequality is proved similarly.
2
Example: Back to the widget example. What can be said about the
probability that last least 550 widgets are made, assuming the mean is
500 and the variance is 100?
Answer:
σ2 100
P [X ≥ 550] = P [X ≥ 500 + 50] ≤ 2 = = 0.0384
σ + 502 100 + 2500
If we only use the different forms of the Markov inquality we get
E[X] 500
P [X ≥ 550] ≤ = = 0.909
550 550
and
E[X 2] σ 2 + µ2
P [X ≥ 550] ≤ 2
= 2
= 0.827
550 550
Probability Inequalities 14
Note that if the production was normally distributed, P [X ≥ 550] =
0.000000287
These probability bounds may not be useful as they may give values greater
than 1. For example, if µ = 500, the Markov bound for
500
P [X ≥ 400] ≤ = 1.25
400
This is a reason why different bounds have been developed. Generally, the
stronger the assumptions you make, the tighter the bounds you can get.
Probability Inequalities 15