Module 14: Expected Values of Sums of Random Variables, ˜

Covariance and Correlation

Module Objectives

Upon completion of this lesson, you should be able to:

1. Use the linearity of the expectation to calculate expectations
2. Recognize the Cauchy-Schwartz inequality
3. Calculate covariances
4. Recognize negative and positive correlation
5. Use covariances to calculate variances
6. Compute the correlation between random variables

Key Terms

expectation Cauchy-Schwartz uncorrelated

positively correlated inequality covariance
correlation negatively correlated

Reading Assignment

Ghahramani, Chapter 10: Sections 10.1, 10.2, and 10.3

Module Introduction
The linearity property of expectation for two random variables easily generalizes to the
multivariable case.

Theorem 10.1 Let X1, X2, . . ., Xn be random variables defined on a common sample space.
Then

Corollary Let X1, X2, . . ., Xn be random variables defined on a common sample space. Then
E(X1 + X2 + . . . + Xn) = E(X1) + E(X2) + . . . + E(Xn).

Example 1: A game requires a player to roll a die and flip a coin. The die roll is interpreted as a
sum of money to be won. The money won is doubled if the coin lands heads and halved if the
coin lands tails. What is the expected value of the winnings? Let X = 1 if the coin Lands heads
and 0 otherwise and let J be the number rolled on the die.
Example 2: A fair die is rolled 10 times. What is the expected sum of the 10 rolls? Letting Xi be
the number on the face of the ith roll, we get:

Example 3: Suppose that Joe and Susie each independently and randomly choose 3 of 10
objects. Let Xi = 1 if they both choose item i and 0 otherwise. Let Yi = 1 if neither choose item i
and 0 otherwise. Also, let Wi = 1 if exactly one of Joe and Susie choose item i and 0 otherwise.
Define

Note that X + Y + W = 10. We can now calculate:

You can calculate the number chosen by exactly one of Joe or Susan by subtraction:
E(number chosen by exactly one of Susie or Joe) = E(W) = 10 ! .9 ! 4.9 = 4.2.

The following theorem is the discrete analogue of the continuous result from Remark 6.4 in your
Textbook:

Theorem 10.2 Let N be a discrete random variable with a set of value values {1, 2, 3, . . .}. Then

Example 4: Let {X1, X2, X3, . . .} be a sequence of continuous, independent, and identically
distributed random variables. Define N = min{n : X1 ≥ X2 ≥X3 ≥ . . ≥ Xn-1, but X n-1 < Xn}.
Notice that N > i if and only if X1 ≥ X2 ≥ X3 ≥ . . . ≥ Xi. Thus, for i ≥ 2, we have 𝑃(𝑁 ≥ 𝑖) =
1
𝑃(𝑋1 ≥ 𝑋2 ≥ 𝑋3 ≥ . . . ≥ 𝑋𝑛 ) = 𝑖! because the Xi’s are independent and identically
distributed. Thus, by Theorem 10.2 we get:
This is simply one of the many strange places the number e appears in mathematics!
Theorem 6.3 (Cauchy-Schwartz Inequality) For random variables X and Y,

Corollary For a random variable X, we have:

Recall that the variance of the random variable X is a measure of the dispersal of the random
variable about its expectation. Suppose X and Y are two jointly distributed random variables.
Then Var(X) and Var(Y) measure the dispersions of X and Y independently, not jointly. To get a
measure of the joint dispersal for the random variables, we consider Var(aX + bY). The text
verifies that Var(aX + bY) = a2Var(X) + b2Var(Y) + 2abE[(X - E(X))(Y - E(Y))].

Since the variances of X and Y are known, if we knew the value of E[(X - E(X))(Y - E(Y))], we
would be able to measure the joint dispersion of the random variables.

Definition If X and Y are jointly distributed random variables, we define the covariance of X and
Y as Cov(X, Y) = E[(X - E(X))(Y - E(Y))].

The text observes that the Cauchy-Schwartz inequality implies Cov(X, Y) # σXσY. Thus, if the
standard deviations of the random variables are finite then the covariance will also be finite.
Theorem 10.4 Let a and b be real numbers. For random variables X and Y:

Var(aX + bY) = a2Var(X) + b2Var(Y) + 2abCov(X, Y),

Var(X + Y) = Var(X) + Var(Y) + 2Cov(X, Y), and
Var(X ! Y) = Var(X) + Var(X) + Var(Y) ! 2Cov(X, Y).

A more useful form of the covariance can be computed. The text verifies that
Cov(X, Y) = E(X, Y)! E(X)E(Y) and Cov(aX + b, cY + d) = acCov(X, Y) for real a, b, c, d.

If the covariance of X and Y is positive, then we say the random variables are positively
correlated. In this case, as one random variable increases, the other increases as well. If the
covariance is negative, they are said to be negatively correlated. In the case of negative
correlation, as one variable increases, the other tends to decrease. They are uncorrelated if the
covariance is zero. If two random variables are independent then they are necessarily
uncorrelated but two random variables which are dependent might be uncorrelated as well.
The generalization of the joint variance to the multidimensional case is straightforward. Let X1,
X2, .. ., Xn be random variables, then:

Example 3: Let X be the number of 1’s and Y be the number of 2’s that are obtained after n rolls
of a fair die. Let:

And also let:

Then we can find the covariance:

From here we can get the covariance of X and Y:

Example 4: A die is rolled twice. Let X represent the sum of the two faces and let Y represent the
first roll minus the second. Let Wi, i = 1, 2 denote the ith outcome. Then:

Note that while the covariance is 0, these variables need not be independent. For example, if I
know that X is odd, that does tell me Y cannot be even.
Although the covariance details how the random variables X and Y vary together, it does depend
upon the units used to measure the random variables. We can standardize the variables and
compute a slightly different measure called the correlation.

Definition Let X and Y be two random variables with finite variances. The covariance between
the standardized X and Y is called the correlation and is denoted by:

The correlation provides the same information as the covariance but has the additional benefit of
being independent of the units involved. The correlation can assume values between !1 and 1. A
value of ρ (X ,Y) = ±1 means the variables X and Y are perfectly linearly related. If the correlation
is 0, then the variables are uncorrelated. The text verifies the following theorem which
summarizes these statements.

Theorem 10.5 For random variables X and Y with correlation coefficient ρ(X ,Y):

Note that correlation only measures linear relationship between variables. There could be a non-
linear relationship (for example, Y = 𝑋 2 ) that is very strong and yet the correlation could be zero.

Module Assignment
Work the following problems from the textbook and submit them to the M14 Assignment in
Moodle.

Ghahramani, Section 10.1 p. 412–14, problems 1–3, 5, 6, 9, 12, 13, 19

Section 10.2 p. 424–27, problems 1, 2, 5, 6–8, 10, 11, 13, 18
Section 10.3 p. 433–34, problems 1–3, 5