Mean (Expectation)
Definition: The mean (expected value) of a random variable measures
its central tendency. In general, the expectation represents the
long-run average value of X .
• For a discrete random variable X with PMF pX (x):
X
E [X ] = xpX (x).
x
• For a continuous random variable X with PDF fX (x):
Z 1
E [X ] = xfX (x)dx.
1
63
Mean (Expectation)
Definition: The mean (expected value) of a random variable measures
its central tendency. In general, the expectation represents the
long-run average value of X .
• For a discrete random variable X with PMF pX (x):
X
E [X ] = xpX (x).
x
• For a continuous random variable X with PDF fX (x):
Z 1
E [X ] = xfX (x)dx.
1
Expectation is essentially a weighted average of all possible values that
the random variable can take, with the probabilities or densities as
weights.
63
Some Key Properties of Expectation
• Linearity: E [aX + bY ] = aE [X ] + bE [Y ], where X and Y are two
random variables and a and b are constants.
Proof Hint: Use the definition of expectation and unless otherwise stated,
random variables X can be both either discrete or continuous. 64
Some Key Properties of Expectation
• Linearity: E [aX + bY ] = aE [X ] + bE [Y ], where X and Y are two
random variables and a and b are constants.
• Expectation of a Constant: E [c] = c.
Proof Hint: Use the definition of expectation and unless otherwise stated,
random variables X can be both either discrete or continuous. 64
Some Key Properties of Expectation
• Linearity: E [aX + bY ] = aE [X ] + bE [Y ], where X and Y are two
random variables and a and b are constants.
• Expectation of a Constant: E [c] = c.
• Non-Negativity: If X 0, then E [X ] 0.
Proof Hint: Use the definition of expectation and unless otherwise stated,
random variables X can be both either discrete or continuous. 64
Some Key Properties of Expectation
• Linearity: E [aX + bY ] = aE [X ] + bE [Y ], where X and Y are two
random variables and a and b are constants.
• Expectation of a Constant: E [c] = c.
• Non-Negativity: If X 0, then E [X ] 0.
• Expectation of Indicator Variables: E [IA ] = P(A).
Proof Hint: Use the definition of expectation and unless otherwise stated,
random variables X can be both either discrete or continuous. 64
Some Key Properties of Expectation
• Linearity: E [aX + bY ] = aE [X ] + bE [Y ], where X and Y are two
random variables and a and b are constants.
• Expectation of a Constant: E [c] = c.
• Non-Negativity: If X 0, then E [X ] 0.
• Expectation of Indicator Variables: E [IA ] = P(A).
• Expectation of Functions of X :
• Let g (x) is any measurable function of x, for example g (x) = x,
g (x) = x 2 , or g (x) = E(Y | X = x). Then
• For discrete random variables X with PMF pX (x),
X
E [g (X )] = g (x) pX (x).
x
• For continuous random variables X with PDF fX (x),
Z 1
E [g (X )] = g (x) fX (x) dx.
1
Proof Hint: Use the definition of expectation and unless otherwise stated,
random variables X can be both either discrete or continuous. 64
Homework 2: Exercise 8 (Jensen’s Inequality)
Prove that
(7.a) If g (x) is a convex function, then:
E [g (X )] g (E [X ])
(7.b) If g (x) is concave, then the inequality is reversed:
E [g (X )] g (E [X ])
Examples (this is only for your understanding, no need to solve
it):
• If g (x) = x 2 (convex), then (E [X ])2 E [X 2 ].
• If g (x) = log(x) (concave), then log E [X ] E [log X ].
65
Expectation of Joint Random Variables
The joint expectation of two random variables X and Y is given by:
• For Discrete Random Variables:
XX
E [XY ] = xy pXY (x, y ).
x y
where pXY (x, y ) is the joint probability mass function (PMF).
66
Expectation of Joint Random Variables
The joint expectation of two random variables X and Y is given by:
• For Discrete Random Variables:
XX
E [XY ] = xy pXY (x, y ).
x y
where pXY (x, y ) is the joint probability mass function (PMF).
• For Continuous Random Variables:
Z 1Z 1
E [XY ] = xy fXY (x, y ) dx dy .
1 1
where fXY (x, y ) is the joint probability density function (PDF).
66
Conditional Expectation
Definition: The conditional expectation of Y given X = x is:
X
E [Y | X ] = ypY |X (y | x). (Discrete)
y
Z 1
E [Y | X ] = yfY |X (y | x)dy . (Continuous)
1
67
Conditional Expectation
Definition: The conditional expectation of Y given X = x is:
X
E [Y | X ] = ypY |X (y | x). (Discrete)
y
Z 1
E [Y | X ] = yfY |X (y | x)dy . (Continuous)
1
Key Properties:
• Linearity:
E [aY + b | X ] = aE [Y | X ] + b.
67
Conditional Expectation
Definition: The conditional expectation of Y given X = x is:
X
E [Y | X ] = ypY |X (y | x). (Discrete)
y
Z 1
E [Y | X ] = yfY |X (y | x)dy . (Continuous)
1
Key Properties:
• Linearity:
E [aY + b | X ] = aE [Y | X ] + b.
• Law of Iterated Expectations:
E [Y ] = E [E [Y | X ]].
Proof Hint:
• Think of E [Y | X ] as a function of X , say g (X ).
• The expectation of Y can be computed in two steps:
1. First, find E [Y | X ], which gives a simpler function of X .
2. Then, take the expectation over X , i.e., E [E [Y | X ]].
67
Variance
• Definition: Variance measures the spread of a random variable
around its mean.
Var(X ) = E [(X E [X ])2 ]
68
Variance
• Definition: Variance measures the spread of a random variable
around its mean.
Var(X ) = E [(X E [X ])2 ]
• Alternative Formula:
Var(X ) = E [X 2 ] (E [X ])2
68
Variance
• Definition: Variance measures the spread of a random variable
around its mean.
Var(X ) = E [(X E [X ])2 ]
• Alternative Formula:
Var(X ) = E [X 2 ] (E [X ])2
• Standard Deviation:
p
X = Var(X )
68
Properties of Variance
Key Properties: (Proof Hint: Use the definition of alternative formula
for variance)
• Variance of a Constant: Var(c) = 0.
69
Properties of Variance
Key Properties: (Proof Hint: Use the definition of alternative formula
for variance)
• Variance of a Constant: Var(c) = 0.
• Scaling Property: Var(aX + b) = a2 Var(X ).
69
Properties of Variance
Key Properties: (Proof Hint: Use the definition of alternative formula
for variance)
• Variance of a Constant: Var(c) = 0.
• Scaling Property: Var(aX + b) = a2 Var(X ).
• Additivity for Independent Variables:
Var(X + Y ) = Var(X ) + Var(Y ).
69
Properties of Variance
Key Properties: (Proof Hint: Use the definition of alternative formula
for variance)
• Variance of a Constant: Var(c) = 0.
• Scaling Property: Var(aX + b) = a2 Var(X ).
• Additivity for Independent Variables:
Var(X + Y ) = Var(X ) + Var(Y ).
• General Variance of a Sum:
Var(X +Y ) = Var(X )+Var(Y )+2Cov(X , Y ) = E (XY ) E [X ]E [Y ].
where Cov(X , Y ) = E [XY ] E [X ]E [Y ] is called the covariance
between the random variables X and Y .
69
Example: Rolling a Fair Die
Random Variable: X represents the outcome of rolling a fair six-sided
die.
Mean Calculation:
6
X 1
E [X ] = x · = 3.5
x=1
6
Variance Calculation:
6
X 1 91
E [X 2 ] = x2 · =
x=1
6 6
91
Var(X ) = E [X 2 ] (E [X ])2 = 3.52 = 2.92
6
70
Example: Uniform Distribution on [0,1]
PDF: fX (x) = 1 for x 2 [0, 1], 0 otherwise.
Mean Calculation:
Z 1
1
E [X ] = x · 1 dx =
0 2
Variance Calculation:
Z 1
1
E [X 2 ] = x 2 · 1 dx =
0 3
✓ ◆2
1 1 1
Var(X ) = E [X 2 ] (E [X ])2 = =
3 2 12
71
Key Takeaways
• Expectation provides a measure of central tendency.
• Variance quantifies the spread and dispersion.
• Key formulas:
P R
• E [X ] = x xpX (x) (Discrete) or E [X ] = xfX (x)dx (Continuous).
• Var(X ) = E [X 2 ] (E [X ])2 .
p
• X = Var(X ).
72
In-Class Activity
Problem: Compute E [X ] and Var(X ) for a fair coin flip where
X 2 {0, 1} and P(X = 1) = 0.5.
73
In-Class Activity
Problem: Compute E [X ] and Var(X ) for a fair coin flip where
X 2 {0, 1} and P(X = 1) = 0.5.
Solution:
The expectation of X is given by:
X
E [X ] = xP(X = x) = 0 · P(X = 0) + 1 · P(X = 1)
x
= 0 · 0.5 + 1 · 0.5 = 0.5.
The variance of X is given by:
Var(X ) = E [X 2 ] (E [X ])2 .
Since, here X 2 = X , we have:
E [X 2 ] = E [X ] = 0.5.
Var(X ) = 0.5 (0.5)2 = 0.5 0.25 = 0.25.
73
Covariance and Correlation: Definitions
Covariance: The covariance between two random variables X and Y is
given by:
Cov(X , Y ) = E [(X E [X ])(Y E [Y ])]
= E [XY ] E [X ]E [Y ].
Properties of Covariance:
• Cov(X , Y ) > 0 (Positive association)
• Cov(X , Y ) < 0 (Negative association)
• Cov(X , Y ) = 0 (Uncorrelated)
• Cov(X , X ) = Var(X )
74
Correlation Coefficient
Definition: The correlation coefficient ⇢(X , Y ) is:
Cov(X , Y )
⇢(X , Y ) = p .
Var(X )Var(Y )
Properties:
• 1 ⇢(X , Y ) 1
• ⇢(X , Y ) = 1 (Perfect positive correlation)
• ⇢(X , Y ) = 1 (Perfect negative correlation)
• ⇢(X , Y ) = 0 (No linear relationship)
75
Example: Covariance Between Two Dice
Let X and Y be the numbers appearing on the first and second rolls of a
fair six-sided die.
6
1X 1 6(7) 7
E [X ] = E [Y ] = i= ⇥ = .
6 6 2 2
i=1
6 6
1 XX 1 441 49
E [XY ] = ij = (21 ⇥ 21) = = .
36 36 36 4
i=1 j=1
Cov(X , Y ) = E [XY ] E [X ]E [Y ].
✓ ◆
49 7 7 49 49
= ⇥ = = 0.
4 2 2 4 4
Thus, X and Y are uncorrelated.
76
Moment Generating Function
(MGF)
Moment Generating Function (MGF)
The MGF of a random variable X is defined as:
(R 1
tx
tX 1
e fX (x) dx, when X is a continuous,
MX (t) = E [e ] = P
tx
x e pX (x), when X is a discrete,
provided the integral and the summations exists.
77
Moment Generating Function (MGF)
The MGF of a random variable X is defined as:
(R 1
tx
tX 1
e fX (x) dx, when X is a continuous,
MX (t) = E [e ] = P
tx
x e pX (x), when X is a discrete,
provided the integral and the summations exists.
• The MGF uniquely determines the distribution of X (when it exists
around t = 0) and can be used to compute the moments of X .
• For example, the n th moments of X are obtained by di↵erentiating
the MGF and evaluating at t = 0:
(n)
E [X n ] = MX (0),
(n)
where MX (t) denotes the n-th derivative of MX (t).
77
Moment Generating Function (MGF)
The MGF of a random variable X is defined as:
(R 1
tx
tX 1
e fX (x) dx, when X is a continuous,
MX (t) = E [e ] = P
tx
x e pX (x), when X is a discrete,
provided the integral and the summations exists.
• The MGF uniquely determines the distribution of X (when it exists
around t = 0) and can be used to compute the moments of X .
• For example, the n th moments of X are obtained by di↵erentiating
the MGF and evaluating at t = 0:
(n)
E [X n ] = MX (0),
(n)
where MX (t) denotes the n-th derivative of MX (t).
For the first two moments:
• Mean (First Moment): E [X ] = MX0 (0).
• Second Moment: E [X 2 ] = MX00 (0).
77
Example: MGF i
Let X be an exponential random variable with the PDF is:
(
x
e , x 0,
fX (x) =
0, otherwise.
Then
(a) Find the moment generating function MX (t).
(b) Find the variance of random variable X using the definitions of MGF.
(c) Also calculate the variance of X using the regular formula of
expectation and variances.
78
Example: MGF ii
(a)
Z 1 Z 1
MX (t) = e tx e x
dx = e x(t )
dx = , for t < .
0 0 t
(b) Therefore expectation,
1
E [X ] = MX0 (0) = , as MX0 (t) =
( t)2
2 2
2
E [X ] = MX00 (0) = 2
, as MX00 (t) =
( t)3
Therefore,
2 2 2 1 1
Var(X ) = E [X ] (E [X ]) = 2 2
= 2
..
79
Example: MGF iii
(c)
Var(X ) = E [X 2 ] (E [X ])2
Z 1 ✓Z 1 ◆2
= x 2 e x dx x e x
dx
0 0
2 1 1
= 2 2
= 2
.
80