AERO 489/689

Foundations of
Aerospace
Autonomy
Dr. Daniel Selva dselva@[Link]
Multiple Random variables
November 12-14 2024

1
Plan for this week
• So far, we have dealt with single random variables. Now we’ll see N
random variables, starting with N=2
• Joint, conditional, marginal PMF, PDF, CDF
• Expected values involving 2+ random variables
• Correlation, covariance, corr. Coeff.
• Independence
• Jointly Gaussian variables
• Transformations of pairs of random variables
• IID random variables
• Estimation
• Main theorems: law of large numbers, central limit theorem
2
Pairs of random variables

3
Joint CDF of 2 random variables
• Joint CDF of a pair of random variables X,Y is 𝐹𝑋,𝑌 𝑥, 𝑦 =
Pr 𝑋 ≤ 𝑥, 𝑌 ≤ 𝑦
• Joint CDFs satisfy the same properties as single-RV CDFs. 𝑦2
• 𝐹𝑋,𝑌 −∞, −∞ = 𝐹𝑋,𝑌 𝑥, −∞ = 𝐹𝑋,𝑌 −∞, 𝑦 = 0; 𝐹𝑋,𝑌 +∞, +∞ = 1 2 1
• 0 ≤ 𝐹𝑋,𝑌 𝑥, 𝑦 ≤ 1
• 𝐹𝑋,𝑌 𝑥, +∞ = 𝐹𝑋 𝑥 ; 𝐹𝑋,𝑌 +∞, 𝑦 = 𝐹𝑌 𝑦 𝑦1
• For 𝑥1 ≤ 𝑥2 , and 𝑦1 ≤ 𝑦2 , 𝐹𝑋,𝑌 𝑥1 , 𝑥2 ≤ 𝐹𝑋,𝑌 (𝑥2 , 𝑦2 ) 3 4
𝑥1 𝑥2
• Pr 𝑥1 ≤ 𝑋 ≤ 𝑥2 , 𝑦1 ≤ 𝑌 ≤ 𝑦2 = 𝐹𝑋,𝑌 𝑥2 , 𝑦2 − 𝐹𝑋,𝑌 𝑥1 , 𝑦2 −
𝐹𝑋,𝑌 𝑥2 , 𝑦1 + 𝐹𝑋,𝑌 𝑥1 , 𝑦1

4
Example: CDF of Jointly Uniform RVs
• Let 𝑋, 𝑌~𝑈𝑛𝑖𝑓( 𝑎, 𝑏 × 𝑎, 𝑏 ). What is the joint CDF?
Joint PDF
• The joint PDF of a pair of random variables is
Pr 𝑥 ≤ 𝑋 ≤ 𝑥 + 𝜀𝑥 , 𝑦 ≤ 𝑌 ≤ 𝑦 + 𝜀𝑦
𝑓𝑋,𝑌 𝑥, 𝑦 = lim
𝜀𝑥 →0,𝜀𝑦 →0 𝜀𝑥 𝜀𝑦
• The following properties hold for joint PDFs:
𝜕2
• 𝑓𝑋,𝑌 𝑥, 𝑦 = 𝐹 𝑥, 𝑦
𝜕𝑥𝜕𝑦 𝑋,𝑌
𝑦 𝑥
• 𝐹𝑋,𝑌 𝑥, 𝑦 = ‫׬‬−∞ ‫׬‬−∞ 𝑓𝑋,𝑌 𝑢, 𝑣 𝑑𝑢𝑑𝑣
• 𝑓𝑋,𝑌 𝑥, 𝑦 ≥ 0, ∀𝑥, 𝑦
∞ ∞
• ‫׬‬−∞ ‫׬‬−∞ 𝑓𝑋,𝑌 𝑢, 𝑣 𝑑𝑢𝑑𝑣=1
𝑦2 𝑥2
• Pr 𝑥1 ≤ 𝑋 ≤ 𝑥2 , 𝑦1 ≤ 𝑌 ≤ 𝑦2 = ‫𝑋𝑓 𝑥׬ 𝑦׬‬,𝑌 𝑢, 𝑣 𝑑𝑢𝑑𝑣
1 1

• More generally: Pr 𝑥, 𝑦 ∈ 𝐴 = ‫𝑋𝑓 𝐴׭‬,𝑌 𝑥, 𝑦 𝑑𝑥𝑑𝑦

6
Marginal distributions
• Given a pair of random variables, the marginal CDF and PDF of X are

𝐹𝑋 𝑥 = 𝐹𝑋,𝑌 𝑥, +∞
∞

𝑓𝑋 𝑥 = න 𝑓𝑋,𝑌 𝑥, 𝑦 𝑑𝑦
−∞

• Note: the two marginals 𝑓𝑋 𝑥 and 𝑓𝑌 𝑦 do not fully specify 𝑓𝑋,𝑌 𝑥, 𝑦 in general.
There are many possible 𝑓𝑋,𝑌 𝑥, 𝑦 that lead to the same marginals!
• Example: For X,Y jointly uniform over [0,1], the marginals are also uniform, but for
X,Y jointly uniform over the unit circle, they are not.

7
Joint PMF
• The joint PMF for a pair of discrete RV X and Y is
𝑃𝑋,𝑌 𝑥, 𝑦 = Pr 𝑋 = 𝑥, 𝑌 = 𝑦 = Pr 𝑋 = 𝑥 ∩ 𝑌 = 𝑦

• The following properties are satisfied

• 0 ≤ 𝑃𝑋,𝑌 𝑥, 𝑦 ≤ 1, ∀𝑥, 𝑦
• σ𝑋 σ𝑌 𝑃𝑋,𝑌 𝑥, 𝑦 = 1
• σ𝑌 𝑃𝑋,𝑌 𝑥, 𝑦 = 𝑃𝑋 (𝑥)

• Let A be a subset of the Cartesian product of the ranges of X and Y. Then:

• Pr 𝑋, 𝑌 ∈ 𝐴 = σ 𝑋,𝑌 ∈𝐴 𝑃𝑋,𝑌 𝑥, 𝑦

8
Conditional CDF, PDF, PMF
• Given 2 discrete RV X and Y, the conditional PMF of X given Y is
Pr 𝑋 = 𝑥, 𝑌 = 𝑦 𝑃𝑋,𝑌 𝑥, 𝑦
𝑃𝑋|𝑌 𝑥|𝑦 = Pr 𝑋 = 𝑥|𝑌 = 𝑦 = =
Pr 𝑌 = 𝑦 𝑃𝑌 (𝑦)

• The conditional PDF of a random variable X given that Y=y is

𝑓𝑋,𝑌 𝑥, 𝑦
𝑓𝑋|𝑌 𝑥|𝑦 =
𝑓𝑌 (𝑦)

• The conditional CDF of X given Y is

𝑥
‫׬‬−∞ 𝑓𝑋,𝑌 𝑢, 𝑦 𝑑𝑢
𝐹𝑋|𝑌 𝑥|𝑦 =
𝑓𝑌 (𝑦)
9
Example: Jointly Gaussian random variables
1 2
• Let X,Y be RV jointly Gaussian with 𝑓𝑋𝑌 𝑥, 𝑦 = exp ቄ− (𝑥 2 −
𝜋 3 3
𝑥𝑦 + 𝑦 2 )ቅ. Calculate 𝑓𝑋 𝑥 and 𝑓𝑋|𝑌 𝑥|𝑦

10
Expected value of g(X,Y)
• Let X,Y be a pair of RV and g(x,y) a 2D function. Then, the expected
value of g(X,Y) is
∞
𝐸 𝑔 𝑋, 𝑌 =ඵ 𝑔 𝑋, 𝑌 𝑓𝑋,𝑌 𝑥, 𝑦 𝑑𝑥𝑑𝑦
−∞

• Example: 𝐸 𝑎𝑋 + 𝑏𝑌 = 𝑎𝐸 𝑋 + 𝑏𝐸 𝑌

11
Correlation, covariance, correlation coefficient
• The correlation between 2 RV is a scalar defined as:
∞
𝑅𝑋,𝑌 = 𝐸 𝑋𝑌 = ඵ 𝑥𝑦 𝑓𝑋,𝑌 𝑥, 𝑦 𝑑𝑥𝑑𝑦
−∞
• If 𝑅𝑋,𝑌 = 0 then X and Y are said to be orthogonal
• The covariance between 2 RV is a scalar defined as:
∞
Cov 𝑋, 𝑌 = 𝐸 𝑋 − 𝜇𝑋 𝑌 − 𝜇𝑌 =ඵ 𝑥 − 𝜇𝑋 𝑦 − 𝜇𝑌 𝑓𝑋,𝑌 𝑥, 𝑦 𝑑𝑥𝑑𝑦
−∞
• If Cov 𝑋, 𝑌 = 0 then X and Y are said to be uncorrelated
• Note that
• Var 𝑋 + 𝑌 = Var X + Var Y + 2Cov 𝑋, 𝑌
• Cov 𝑋, 𝑌 = 𝑅𝑋,𝑌 − 𝜇𝑋 𝜇𝑌

12
Correlation coefficient
• The correlation coefficient between two RV X and Y is
𝐶𝑜𝑣 𝑋, 𝑌
𝜌𝑋𝑌 =
𝜎𝑋 𝜎𝑌
• Note this is a normalized version of covariance so that −1 ≤ 𝜌𝑋𝑌 ≤ 1

• Interpretation
• 𝜌𝑋𝑌 = ±1 if 𝑌 = 𝑎𝑋
• 𝜌𝑋𝑌 = 0 if X and Y are uncorrelated

• Note that X and Y uncorrelated isn’t the same as independent.

13
Example
• Given the joint PMF given below, answer the following questions
P(X,Y) Y=1 Y=2 Y=3 Y=4
X=1 0.10 0.05 0.05 0.05
X=2 0.10 0.02 0.05 0.10
X=3 0.30 0.10 0.03 0.05
1. What is Pr 𝑋 = 1, 𝑌 ≥ 3 ? Pr 𝑋 = 1|𝑌 ≥ 3 ?
• 0.05+0.05=0.1
• 0.1/0.33=0.3
2. What is 𝑃𝑋 (𝑥)? 𝑃𝑌 (𝑦)?
• 𝑃𝑋 𝑥 = σ𝑦 𝑃𝑋,𝑌 𝑥, 𝑦 = 0.25,0.27,0.48
• 𝑃𝑌 𝑦 = σ𝑥 𝑃𝑋,𝑌 𝑥, 𝑦 = 0.50,0.17,0.13,0.20
3. What is 𝑃𝑌|𝑋 (𝑦|𝑋 = 1)?
0.10,0.05,0.05,0.05
• 𝑃𝑌|𝑋 𝑦 𝑋 = 1 = = 0.4,0.2,0.2,0.2
0.10+0.05+0.05+0.05

14
Example
• Given the joint PMF given below, answer the following questions
P(X,Y) Y=1 Y=2 Y=3 Y=4
X=1 0.10 0.05 0.05 0.05
X=2 0.10 0.02 0.05 0.10
X=3 0.30 0.10 0.03 0.05

• What is 𝐸 𝑋 ? 𝐸 𝑌 ?
𝐸 𝑋 = 0.25,0.27,0.48 ⋅ 1,2,3 𝑇 = 2.23 = 𝜇𝑋
𝐸 𝑌 = 0.50,0.17,0.13,0.20 ⋅ 1,2,3,4 𝑇 = 2.03 = 𝜇𝑌
• What is the variance of X? and Y?
𝑣𝑎𝑟 𝑋 = 𝜎𝑋2 = 𝐸 𝑋 − 𝜇𝑋 2 = 0.6771
𝑣𝑎𝑟 𝑌 = 𝜎𝑌2 = 𝐸 𝑌 − 𝜇𝑌 2 = 1.4291

15
Think-pair-share
• Given the joint PMF given below, answer the following questions
P(X,Y) Y=1 Y=2 Y=3 Y=4
X=1 0.10 0.05 0.05 0.05
X=2 0.10 0.02 0.05 0.10
X=3 0.30 0.10 0.03 0.05

• What is the correlation between X and Y?

𝑅𝑋,𝑌 = 𝐸 𝑋𝑌 = 0.10 ⋅ 1 ⋅ 1 + 0.05 ⋅ 1 ⋅ 2 + ⋯ + 0.05 ⋅ 3 ⋅ 4 = 4.3
• What is the covariance between X and Y?
Cov 𝑋, 𝑌 = 𝐸 𝑋 − 𝜇𝑋 𝑌 − 𝜇𝑌 = 𝑅𝑋,𝑌 − 𝜇𝑋 𝜇𝑌 = 4.3 − 2.23 ⋅ 2.03 = −0.2269
• What is the correlation coefficient between X and Y?
𝐶𝑜𝑣 𝑋, 𝑌 −0.2269
𝜌𝑋𝑌 = = = −0.2307
𝜎𝑋 𝜎𝑌 0.6771 ⋅ 1.4291

16
Independence
• Two random variables are independent if
𝐹𝑋𝑌 𝑥, 𝑦 = 𝐹𝑋 𝑥 𝐹𝑌 𝑦 ∀𝑥, 𝑦
• Or equivalently
𝑓𝑋𝑌 𝑥, 𝑦 = 𝑓𝑋 𝑥 𝑓𝑌 𝑦 ∀𝑥, 𝑦; 𝑃𝑋,𝑌 𝑥, 𝑦 = 𝑃𝑋 𝑥 𝑃𝑌 𝑦 ∀𝑥, 𝑦
• Or equivalently
𝑓𝑌|𝑋 𝑦|𝑥 = 𝑓𝑌 𝑦 ∀𝑥, 𝑦; 𝑃𝑌|𝑋 𝑦|𝑥 = 𝑃𝑌 𝑦 ∀𝑥, 𝑦
• Intuitively, knowing the value of X doesn’t change the distribution of Y
• Note that if X and Y are independent, then 𝑔1 𝑋 and 𝑔2 𝑌 are also
independent.
• If X and Y are independent, then they are uncorrelated
• But the converse is not necessarily true!

17
Example
P(X,Y) Y=1 Y=2 Y=3 Y=4
X=1 0.10 0.05 0.05 0.05
X=2 0.10 0.02 0.05 0.10
X=3 0.30 0.10 0.03 0.05

• Are X and Y independent?

• No, because, for example:
0.10,0.05,0.05,0.05
• 𝑃𝑌|𝑋 𝑦 𝑋 = 1 = = 0.4,0.2,0.2,0.2
0.10+0.05+0.05+0.05
0.10,0.02,0.05,0.10
• 𝑃𝑌|𝑋 𝑦 𝑋 = 2 = = 0.37,0.07,0.19,0.37
0.10+0.02+0.05+0.10
• Or equivalently,𝑃𝑋,𝑌 𝑥, 𝑦 ≠ 𝑃𝑋 𝑥 𝑃𝑌 𝑦
• Recall that 𝑃𝑋 𝑥 = 0.25,0.27,0.48 ; 𝑃𝑌 𝑦 = 0.50,0.17,0.13,0.20
• For x=1, y=1, 𝑃𝑋 𝑥 𝑃𝑌 𝑦 = 0.5 ∗ 0.25 = 0.125 ≠ 0.1
18
Jointly Gaussian random variables
• A pair of RV X and Y are jointly Gaussian if their joint PDF can be expressed as
𝑓𝑋,𝑌 𝑥, 𝑦
2
𝑥 − 𝜇𝑋 2 2𝜌𝑋𝑌 𝑥 − 𝜇𝑋 𝑦 − 𝜇𝑌 𝑦 − 𝜇𝑌
− +
1 𝜎𝑋2 𝜎𝑋 𝜎𝑌 𝜎𝑌2
= exp − 2
2𝜋𝜎𝑋 𝜎𝑌 1 − 𝜌𝑋𝑌 2 2 1 − 𝜌𝑋𝑌

• Uncorrelated Gaussian random variables are independent

• Exercise: Show that if X and Y are jointly Gaussian, then X given Y is also Gaussian
𝜎𝑋
• Mean of X given Y: 𝜇𝑋 𝑦 = 𝜇𝑋 + 𝜌𝑋𝑌 𝑦 − 𝜇𝑌
𝜎𝑌
• Variance of X given Y: 𝜎𝑋′ = 𝜎𝑋 1 − 2
𝜌𝑋𝑌

19
Transformations of pairs of RV
• Let 𝑍 = 𝑔 𝑋, 𝑌 . Then
𝐹𝑍 𝑧 = Pr 𝑍 ≤ 𝑧 = Pr 𝑔 𝑋, 𝑌 ≤ 𝑍 = ඵ 𝑓𝑋,𝑌 𝑥, 𝑦 𝑑𝑥𝑑𝑦
𝑔 𝑋,𝑌 ≤𝑍
𝑑𝐹𝑍 𝑧
𝑓𝑍 𝑧 =
𝑑𝑧
• Example: Compute the pdf of the sum of two independent random variables
𝐹𝑍 𝑧 = Pr 𝑔 𝑋, 𝑌 ≤ 𝑍 = ඵ 𝑓𝑋,𝑌 𝑥, 𝑦 𝑑𝑥𝑑𝑦
𝑥+𝑦≤𝑧
+∞ 𝑧−𝑦 +∞
𝐹𝑍 𝑧 = න න 𝑓𝑋 𝑥 𝑓𝑌 𝑦 𝑑𝑥𝑑𝑦 = න 𝐹𝑋 𝑧 − 𝑦 𝑓𝑌 𝑦 𝑑𝑦
−∞ −∞ +∞ −∞
𝑑𝐹𝑍 𝑧
𝑓𝑍 𝑧 = = න 𝑓𝑋 𝑧 − 𝑦 𝑓𝑌 𝑦 𝑑𝑦
𝑑𝑧 −∞

20
Example
P(X,Y) Y=1 Y=2 Y=3 Y=4
X=1 0.10 0.05 0.05 0.05
X=2 0.10 0.02 0.05 0.10
X=3 0.30 0.10 0.03 0.05
• What is the PMF of X+Y? 𝑃𝑍 𝑧 = 𝑃𝑧 𝑋 + 𝑌 = 𝑧
• Intuitively, X+Y can take values from 2 to 7. We can just add the relevant probabilities
• 𝑃𝑍 2 = 𝑃𝑋𝑌 𝑋 = 1, 𝑌 = 1 = 0.10
• 𝑃𝑍 3 = 𝑃𝑋𝑌 𝑋 = 1, 𝑌 = 2 + 𝑃𝑋𝑌 𝑋 = 2, 𝑌 = 1 = 0.05 + 0.10 = 0.15
• …
• 𝑃𝑍 6 = 𝑃𝑋𝑌 𝑋 = 3, 𝑌 = 3 + 𝑃𝑋𝑌 𝑋 = 2, 𝑌 = 4 = 0.03 + 0.10 = 0.13
• 𝑃𝑍 7 = 𝑃𝑋𝑌 𝑋 = 3, 𝑌 = 4 = 0.05

• Check it’s the same as the formula: 𝑃𝑍 𝑧 = σ𝑦 𝑃𝑋 𝑧 − 𝑦 𝑃𝑌 𝑦

21
Extension to N>2 random
variables
IID random variables, estimation, Law of Large Numbers and Central Limit Theorem

22
Joint CDF, PDF, PMF for N random variables
• Joint PMF of N discrete random variables 𝑿 = 𝑋1 , 𝑋2 , … , 𝑋𝑁 is
𝑃𝑿 𝒙 = Pr 𝑋1 = 𝑥1 , … , 𝑋𝑁 = 𝑥𝑁
• Joint CDF of N random variables 𝑿 = 𝑋1 , 𝑋2 , … , 𝑋𝑁 is
𝐹𝑿 𝒙 = 𝐹𝑿 𝑥1 , … , 𝑥𝑁 = Pr 𝑋1 ≤ 𝑥1 , … , 𝑋𝑁 ≤ 𝑥𝑁
• Joint PDF of N random variables 𝑿 = 𝑋1 , 𝑋2 , … , 𝑋𝑁 is
𝜕𝑁
𝑓𝑿 𝒙 = 𝐹𝑿 𝒙
𝜕𝑥1 … 𝜕𝑥𝑁
• Joint PMFs/CDFs/PDFs satisfy the same properties as for N=2.
• 𝐹𝑿 𝒙 is 0 when any 𝑋𝑖 → −∞; it is 1 when all 𝑋𝑖 → ∞, and is non-decreasing w.r.t.
each 𝑋𝑖 .
• Similarly, 𝑓𝑿 𝒙 is non-negative and integrates to 1. 𝑃𝑿 𝒙 is non-negative and sums
up to 1.

23
Marginal and Conditional CDF, PDF, PMF
• Marginals: If you evaluate CDF at +∞ (or integrate the PDF from −∞ to +∞)
for a subset of variables you obtain the marginal CDFs (PDFs respectively)
for the other variables
• Given 𝑿 = 𝑋1 , 𝑋2 , … 𝑋𝑀 , 𝑋𝑀+1 , … , 𝑋𝑁 , the conditional PMF of
𝑋1 , 𝑋2 , … 𝑋𝑀 given 𝑋𝑀+1 , … , 𝑋𝑁 is
𝑃𝑿 𝑥1 , … , 𝑥𝑁
P𝑋1 ,𝑋2 ,…𝑋𝑀 |𝑋𝑀+1 ,…,𝑋𝑁 𝑥1 , … , 𝑥𝑀 |𝑥𝑀+1 , … , 𝑥𝑁 =
𝑃𝑿 (𝑥𝑀+1 , … , 𝑥𝑁 )
• Given 𝑿 = 𝑋1 , 𝑋2 , … 𝑋𝑀 , 𝑋𝑀+1 , … , 𝑋𝑁 , the conditional PDF of
𝑋1 , 𝑋2 , … 𝑋𝑀 given 𝑋𝑀+1 , … , 𝑋𝑁 is
𝑓𝑿 𝑥1 , … , 𝑥𝑁
f𝑋1 ,𝑋2 ,…𝑋𝑀 |𝑋𝑀+1 ,…,𝑋𝑁 𝑥1 , … , 𝑥𝑀 |𝑥𝑀+1 , … , 𝑥𝑁 =
𝑓𝑿 (𝑥𝑀+1 , … , 𝑥𝑁 )

24
Independence
• N random variables are independent iff any subset of them are
independent.
• Necessary and sufficient condition: N random variables are
independent if
𝑁

𝐹𝑿 𝒙 = ෑ 𝐹𝑋𝑖 𝑥𝑖
𝑖=1
• Or equivalently
𝑁

𝑓𝑿 𝒙 = ෑ 𝑓𝑋𝑖 𝑥𝑖
𝑖=1

25
Example
• Example: log-likelihood function of N independent Gaussian random
variables with mean 𝜇𝑖 and variance 𝜎𝑖2 .
log 𝑓𝑿 𝒙 = log ς𝑁 𝑓
𝑖=1 𝑋𝑖 𝑖𝑥 = σ 𝑁
𝑖=1 log 𝑓𝑋𝑖 𝑥𝑖 =
1 𝑥 −𝜇 2
1 −2 𝑖𝜎 𝑖 1
σ𝑁
𝑖=1 log 𝑒 𝑖 = σ𝑁
𝑖=1 log +
2𝜋𝜎𝑖2 2𝜋𝜎𝑖2
1 𝑥𝑖 −𝜇𝑖 2
− 1 𝑁 1 𝑁 𝑥𝑖 −𝜇𝑖 2
σ𝑁
𝑖=1 log 𝑒 2 𝜎𝑖 2
= − σ𝑖=1 log 2𝜋𝜎𝑖 − σ𝑖=1 =
2 2 𝜎𝑖
𝑁 1 𝑁 1 𝑥 −𝜇 2
− log 2𝜋 − σ𝑖=1 log 𝜎𝑖2 − σ𝑁 𝑖 𝑖
2 2 2 𝑖=1 𝜎𝑖

26
IID random variables
• A set or sequence of random variables 𝑋1 , 𝑋2 , … , 𝑋𝑁 is IID if the
random variables are
• Independent: 𝐹𝑋1 …𝑋𝑁 𝑥1 , … , 𝑥𝑁 = ς𝑖 𝐹𝑋𝑖 𝑥𝑖
• Identically distributed: 𝐹𝑋𝑖 𝑥𝑖 = 𝐹𝑋 𝑥 , ∀𝑖

• Examples:
• Flip a coin N times
• Perform N measurements of the temperature in a room
• Receive N symbols in a digital communication system
• Test N light bulbs to failure
• Run a stochastic algorithm N times

27
Correlation and covariance matrices
• Let X be a vector of N random variables 𝑿 = 𝑋1 , 𝑋2 , … , 𝑋𝑁 𝑇 . Then, the expected value of 𝑿 is a
vector
𝑇 𝑇
𝐸 𝑿 = 𝝁𝑿 = 𝐸 𝑋1 , … , 𝐸 𝑋𝑁 = 𝜇𝑋1 , … , 𝜇𝑋𝑁
• The correlation matrix of N RV is a matrix defined as:
𝐸 𝑋12 ⋯ 𝐸 𝑋1 𝑋𝑁
𝑹𝑿𝑿 = 𝐸 𝑿𝑿𝑻 = ⋮ ⋱ ⋮
𝐸 𝑋𝑁 𝑋1 ⋯ 𝐸 𝑋𝑁2
• If 𝑹𝑿𝑿 is diagonal, then the random variables are orthogonal
• The covariance matrix between N RV is a matrix defined as :
𝑣𝑎𝑟 𝑋1 ⋯ 𝑐𝑜𝑣 𝑋1 𝑋𝑁
𝑪𝑿𝑿 = 𝐸 𝑿 − 𝝁𝑿 𝑿 − 𝝁𝑿 𝑻 = ⋮ ⋱ ⋮
𝑐𝑜𝑣 𝑋𝑁 𝑋1 ⋯ 𝑣𝑎𝑟 𝑋𝑁
• If 𝑪𝑿𝑿 is diagonal, then the random variables are uncorrelated
• The correlation and covariance matrices are square, symmetric, and positive definite.

28
Example
• Ley 𝒀 = 𝐴𝑋 + 𝒃 where X,Y are vectors of N random variables, A is an
𝑁 × 𝑁 matrix and 𝒃 is an N-dimensional vector.
• Compute 𝑅𝑌𝑌 and 𝐶𝑌𝑌 .

29
Estimation and detection
• Widely applicable problems with applications in autonomous systems,
statistics, machine learning, GN&C, …
• Prediction: Predict 𝑌 as a function of 𝑋1 … 𝑋𝑁 or Y = 𝑋𝑡+1 as a function of 𝑋1 … 𝑋𝑡
• Interpolation: Given 𝑋1 … 𝑋𝑡 estimate some intermediate value
• Filtering: Assuming 𝑿 = 𝒀 + 𝑵, where N is noise, estimate 𝒀
• Estimation: Given some observations of X, estimate its pdf (typically assuming
𝑓𝑿 𝒙; 𝜽 and estimating the values of the parameters 𝜽
• Detection/hypothesis testing: Decide between a number of discrete hypotheses Y ∈
𝑦1 … 𝑦𝑁 based on observations 𝑋1 … 𝑋𝑁 .
• In ML, we distinguish between regression and classification problems,
depending on whether the quantity to predict is continuous or discrete
respectively.

30
MAP detection problem
• M hypotheses: 𝑌 ∈ 𝑦1 … 𝑦𝑀
• N observations: 𝑋 = 𝑋1 , … , 𝑋𝑁
• Problem: Decide the most likely hypothesis 𝑌෠ = 𝑦𝑘 based on the observations
• Approach: 𝑌෠ = 𝑎𝑟𝑔 max Pr 𝑌 = 𝑦𝑘 |𝑿 = 𝒙 (posterior probability)
𝑘
• Note that
Pr 𝑿 = 𝒙|𝑌 = 𝑦𝑘 𝑃𝑟 𝑌 = 𝑦𝑘
Pr 𝑌 = 𝑦𝑘 |𝑿 = 𝒙 =
σ𝑘 Pr 𝑿 = 𝒙|𝑌 = 𝑦𝑘 𝑃𝑟 𝑌 = 𝑦𝑘
• And the denominator does not depend on k (it’s 𝑓𝑿 (𝒙)). Thus, MAP estimation can
be expressed as maximizing the product of the prior and the likelihood function:
𝑌෠ = 𝑎𝑟𝑔 max Pr 𝑿 = 𝒙|𝑌 = 𝑦𝑘 𝑃𝑟 𝑌 = 𝑦𝑘
𝑘

31
MLE estimation
• For Y continuous, the MAP decision rule is
𝑌෠ = 𝑎𝑟𝑔 max 𝑓 𝑦|𝒙 = 𝑎𝑟𝑔 max 𝑓 𝒙|𝑦 𝑓 𝒚
𝑦 𝑦

• Often, we can assume that the prior on Y is uniform. In this case the MAP
reduces to maximizing the likelihood function
𝑌෠ = 𝑎𝑟𝑔 max 𝑓 𝑦|𝒙 = 𝑎𝑟𝑔 max 𝑓𝑿|𝑌 𝒙|𝑦 𝑓𝑌 (𝑦) = 𝑎𝑟𝑔 max 𝑓𝑿|𝑌 𝒙|𝑦
𝑦 𝑦 𝑦

• Oftentimes, it is convenient to maximize the log-likelihood function instead

(e.g., jointly Gaussian). We can do that because log is a monotonically
increasing function.
32
Example in 1D: Digital communication system
• A digital communication system works by sending 2 pulses of different
amplitude: 𝑦 = 𝑦1 = +1 for a ‘1’ and 𝑦 = 𝑦0 = −1 for a ‘0’.
• The channel adds additive Gaussian noise with mean 0 and variance
𝜎 2 to the transmitted amplitude received: 𝑥 = 𝑦 + 𝑛
• What is the MAP decision rule for the receiver assuming 0s and 1s are
equally likely?

33
Example in 1D: Digital communication system
• A digital communication system works by sending 2 pulses of different amplitude: 𝑦 = 𝑦1 = +1 for a ‘1’ and 𝑦 = 𝑦0 = −1 for a ‘0’.
• The channel adds additive Gaussian noise with mean 0 and variance 𝜎 to the transmitted amplitude received: 𝑥 = 𝑦 + 𝑛
• What is the MAP decision rule for the receiver assuming 0s and 1s are equally likely?

𝑌෠ = 𝑎𝑟𝑔 max f 𝑥|𝑌 = 𝑦𝑘 𝑃𝑟 𝑌 = 𝑦𝑘

𝑘
PX|Y 𝑋|𝑌 = 𝑦𝑘 ~𝑁 𝜇 = 𝑦𝑘 ; 𝜎

𝑓 𝑥|𝑌=𝑦1 Pr 𝑋=𝑥|𝑌=𝑦1
• Given we received x: detect a 1 if > 1 or log >0
𝑓 𝑥|𝑌=𝑦0 Pr 𝑋=𝑥|𝑌=𝑦0

1 𝑥−𝜇1 2
1 −
𝑒 2 𝜎
𝑓 𝑥|𝑌 = 𝑦1 2𝜋𝜎 2 1 𝑥 − 𝜇1 2 1 𝑥 − 𝜇0 2
= = exp − + >1
𝑓 𝑥|𝑌 = 𝑦0 1 𝑥−𝜇0 2 2 𝜎 2 𝜎
1 −
2
𝑒 2 𝜎
2𝜋𝜎
𝑓 𝑥|𝑌 = 𝑦1 1 𝑥 − 𝜇1 2 1 𝑥 − 𝜇0 2
log =− + >0
𝑓 𝑥|𝑌 = 𝑦0 2 𝜎 2 𝜎
1
𝑥 − 𝜇0 2 > 𝑥 − 𝜇1 2 ⇒ 2𝑥 𝜇0 − 𝜇1 > 𝜇02 − 𝜇12 ⇒ 𝑥 > 𝜇0 + 𝜇1
2

34
MMSE estimation
• Problem: Given some observations 𝑿 and Y, estimate or predict 𝑌 as a function of X
𝑌෠ = 𝑔(𝑿) ≈ 𝑌

• Approach: Find a function that minimizes mean square error between the observed and predicted
values:
2
𝑌෠ = arg min 𝐸 𝑌෠ − 𝑌 |𝑿
𝒈

• Solution: One can show that the best function is

𝑌෠ = 𝑔 𝑿 = 𝐸 𝑌|𝑿 = 𝒙
2
• Proof: 𝑀𝑆𝐸 = 𝐸 𝑌෠ − 𝑌 |𝑿 = 𝒙 = 𝐸 𝑔 𝑿 − 𝑌 2 |𝑿 = 𝒙
Let 𝑔 𝒙𝟎 = 𝑦0 . Then 𝑀𝑆𝐸 𝒙𝟎 = 𝐸 𝑦0 − 𝑌 2 |𝑿 = 𝒙𝟎 = 𝐸 𝑦02 + 𝑌 2 − 2𝑦0 𝑌|𝑿 = 𝒙𝟎
𝑑𝑀𝑆𝐸 𝑥0
= 2𝑦0 − 2𝐸 𝑌 𝑿 = 𝒙𝟎 = 0 → 𝑦0 = 𝐸 𝑌 𝑿 = 𝒙𝟎
𝑑𝑦0

35
MMSE under linear models
• Often, we assume the function can be approximated by a parameterized family of functions (e.g., linear
models, polynomials)
𝑔 𝑿 = 𝑔(𝑿; 𝜽)
෡ = arg min 𝐸 𝑔(𝑿; 𝜽) − 𝑌 2 |𝑿 = 𝒙
𝒀
𝜽
• The most common restriction is
𝑔 𝑿; 𝒂 = 𝑎1 𝑋1 + ⋯ + 𝑎𝑁 𝑋𝑁 = 𝑿𝑇 𝒂
• In this case, for 𝑿 = 𝒙:
2
𝑀𝑆𝐸 = 𝐸 𝑎1 𝑋1 + ⋯ + 𝑎𝑁 𝑋𝑁 − 𝑌
• We differentiate wr.t. 𝑎𝑘 :
𝜕𝑀𝑆𝐸 𝒙
= 2𝐸 𝑋𝑘 𝑎1 𝑋1 + ⋯ + 𝑎𝑁 𝑋𝑁 − 𝑌 = 0
𝜕𝑎𝑘
𝐸 𝑋𝑘 𝑌 = 𝑎1 𝐸 𝑋𝑘 𝑋1 + 𝑎2 𝐸 𝑋𝑘 𝑋2 + ⋯ + 𝑎𝑁 𝐸 𝑋𝑘 𝑋𝑁
• Grouping all N equations, and recalling the definition of 𝑅𝑋𝑋 , we find that the optimal values of a are
𝒂∗ = 𝑹−𝟏𝑿𝑿 𝐸 𝑿𝑌

36
Estimating the Mean of IID random variables
• Let 𝑋𝑖 be IID with some underlying common pdf 𝑓𝑋 𝑥 such that E 𝑋 = 𝜇𝑋
• How do we estimate 𝜇𝑋 from the sequence of observed 𝑋𝑖 ?
𝜇ෞ𝑋 = 𝑔 𝑋1 , 𝑋2 , … , 𝑋𝑁
• We would like the estimator to be unbiased:
𝐸 𝜇ෞ𝑋 = 𝜇𝑋
• We would like to be efficient, i.e., to minimize the error, e.g.:
min 𝐸 𝜇ෞ𝑋 − 𝜇𝑋 2
𝑔
• Example (BLUE): The sample mean is the Best Linear Unbiased Efficient estimator
of the mean of IID random variables (proof in pp 291-292)
1
𝜇ෞ𝑋 = ෍ 𝑥𝑖 = 𝑥ҧ
𝑁
𝑖

37
Think-pair-share
• Let 𝑋𝑖 be IID with some underlying common pdf 𝑓𝑋 𝑥 such that
E 𝑋 = 𝜇𝑋
• Consider a linear estimator of the mean
𝜇ෞ𝑋 = 𝑎1 𝑋1 + ⋯ + 𝑎𝑁 𝑋𝑁
• What condition do the coefficients have to satisfy so that the
estimator is unbiased?

𝐸 𝜇ෞ𝑋 = 𝐸 𝑎1 𝑋1 + ⋯ + 𝑎𝑁 𝑋𝑁 = 𝑎1 𝐸 𝑋1 + ⋯ + 𝑎𝑁 𝐸 𝑋𝑁 =
𝜇𝑋 σ𝑖 𝑎𝑖 = 𝜇𝑋 ⇒ σ𝑖 𝑎𝑖 = 1

38
ML Estimator of the mean of IID RV
• We can also adopt the MLE approach
𝜇ො𝑀𝐿𝐸 = arg max 𝑓𝑋|𝜇 𝑋1 , … , 𝑋𝑁 |𝜇
𝜇

• Example: What is the MLE of the mean of a sequence of n IID Gaussian random variables N(𝜇, 𝜎 2 )?
1 1
𝑓𝑿|𝜇 𝒙|𝜇 = 2 𝑛/2
exp − 2 ෍ 𝑥𝑖 − 𝜇 2
2𝜋𝜎 2𝜎
𝑖
𝜇∗ = arg max log 𝑓𝑿 𝒙|𝜇 = arg max log σ𝑖 𝑥𝑖 − 𝜇 2
𝜇 𝜇
𝑑 2
෍ 𝑥𝑖 − 𝜇 = −2 ෍ 𝑥𝑖 − 𝜇 = 0
𝑑𝜇
𝑖 𝑖
∗
1
𝜇 = ෍ 𝑥𝑖
𝑛
𝑖

39
Variance of the sample mean estimator
1
• Consider the sample mean estimator 𝜇ෞ𝑋 = σ𝑖 𝑥𝑖
𝑁
• We know that it’s unbiased 𝐸 𝜇ෞ𝑋 = 𝜇𝑋

• What about its variance?

2
2
1 𝜎𝑋2
𝐸 𝜇ෞ𝑋 − 𝜇𝑋 =𝐸 ෍ 𝑥𝑖 − 𝜇𝑋 …=
𝑁 𝑁
𝑖

• Gets smaller as we receive more observations (i.e., we can average out

stochastic error given enough IID observations)

40
Estimating the variance of IID RV
• Let 𝑋 be IID with some underlying common pdf 𝑓 𝑥 such that E 𝑋 − 𝜇 2 =
𝑖 𝑋 𝑋
𝜎𝑋2
• How do we estimate 𝜎𝑋2 from the sequence X?
𝜎
ෞ𝑋 = 𝑔 𝑋1 , 𝑋2 , … , 𝑋𝑁
• Still want our estimator to be unbiased and efficient, but probably not linear ➔
Take MLE approach
• Example: MLE estimator of the variance for N IID Gaussian RV with mean 𝜇:
𝑑 𝑑 1 1 2
𝑓𝑋|Σ 𝑥|𝜎 = 𝑁 exp − 2
෍ 𝑥𝑖 − 𝜇 =0
𝑑𝜎 𝑑𝜎 2𝜎
2𝜋𝜎 2 2 𝑖

෢2
1
𝜎𝑀𝐿 = ෍ 𝑥𝑖 − 𝜇 2
𝑁
𝑖

41
Estimating the variance of IID RV
• An estimator that doesn’t require knowing the true mean?
• Consider
1 2 1
𝜎෢2 = σ𝑖 𝑥𝑖 − 𝑥ҧ with 𝑥ҧ = σ 𝑥
𝑁 𝑁 𝑖 𝑖
• This seems like a reasonable choice; however, this turns out to be a biased estimator!
෢ 𝑁−1 2
2
𝐸 𝜎 =⋯= 𝜎 ≠ 𝜎2
𝑁
• So, it consistently underestimates the true variance.
• Intuition: The sample mean is the point that minimizes the average distance from the sample (it minimizes
MSE) but the distance to the true mean will be slightly larger.
• Therefore, the following unbiased estimator is used instead
1
𝜎෢2 = ෍ 𝑥𝑖 − 𝜇ො 2
𝑁−1
𝑖
• This is the sample variance (a.k.a. Bessel correction).

42
Estimating the CDF of IID RV
• Let 𝑋𝑖 be IID with common CDF 𝐹𝑋𝑖 𝑥𝑖 = 𝐹𝑋 𝑥 = Pr 𝑋 ≤ 𝑥
• How do we estimate 𝐹𝑋 𝑥 from the sequence of observed 𝑋𝑖 ?
1, 𝑋𝑖 ≤ 𝑥
• Let 𝑌𝑖 = ቊ
0, 𝑋𝑖 > 0
• Note the 𝐸 𝑌𝑖 = Pr 𝑋𝑖 ≤ 𝑥 = 𝐹𝑋 𝑥
• Hence, we can estimate 𝐹𝑋 𝑥 using the sample mean of 𝑌𝑖
1 1
෢
𝐹𝑋 𝑥 = ෍ 𝑌𝑖 = ෍ 1 − 𝑢 𝑋𝑖 − 𝑥
𝑁 𝑁
𝑖 𝑖
• This is basically counting how many observations satisfy 𝑋𝑖 ≤ 𝑥 . Intuition:
Histograms are well founded

43
Convergence of random sequences
• Let 𝑋1 , 𝑋2 , … , 𝑋𝑛 be n IID random variables.
1
• Let Sn = σ𝑖 𝑋𝑖 . The sequence 𝑆1 , 𝑆2 , … , 𝑆𝑛 is a sequence of random
𝑛
variables. Does 𝑆1 , 𝑆2 , … , 𝑆𝑛 converge to the true mean? Intuitively we
want
lim 𝑆𝑛 = 𝜇
𝑛→∞
• An estimator satisfying this is called consistent.
• If deterministic: lim 𝑠𝑛 = 𝑠 ⇔ ∀𝜀 > 0, ∃𝑖𝜀 : 𝑠𝑖 − 𝑠 < 𝜀∀𝑖 > 𝑖𝜀
𝑛→∞
• For random sequences, how do we define this?

44
Modes of convergence
• Convergence everywhere: Every realization of the sequence converges to
the same value (e.g., the true mean).
𝑋
• E x: 𝑋𝑛 = converges everywhere to a constant 0.
1+𝑛2
𝑛𝑋
• Ex: 𝑋𝑛 = converges everywhere to the random variable 𝑋.
1+𝑛

• In practice, one or more of the realizations may not converge.

• Convergence almost everywhere: The probability that a realization of the

sequence does not converge is 0.
• Ex: 𝑍𝑛 = 𝑠𝑖𝑛𝑐 𝑛𝑍 for 𝑍~𝑈𝑛𝑖𝑓[0,1) converges to 0 everywhere except for Z=0
where it converges to 1. Clearly Pr(𝑍 = 0) = 0.

45
Modes of convergence
• Convergence in probability: The sequence 𝑆1 , 𝑆2 , … , 𝑆𝑛 converges to S in probability if for any 𝜀 >
0:
lim Pr 𝑆𝑛 − 𝑆 > 𝜀 = 0
1
𝑛→∞
• Example: Sn = σ 𝑋 for 𝑋𝑖 IID Gaussian converges in probability to 𝜇 since lim Pr 𝑆𝑛 − 𝜇 > 𝜀 =
𝑛 𝑖 𝑖 𝑛→∞
𝑛𝜀
lim 2𝑄 𝜎2
=0
𝑛→∞

• Convergence in mean
2
square: The sequence 𝑆1 , 𝑆2 , … , 𝑆𝑛 converges to S in mean square if
lim E 𝑆𝑛 − 𝑆 =0
𝑛→∞ 1
• Example: The same Sn = σ𝑖 𝑋𝑖 converges in mean square to 𝜇 since lim E 𝑆𝑛 − 𝑆 2 = lim 𝜎 2 /𝑛 = 0
𝑛 𝑛→∞ 𝑛→∞
• Convergence in distribution: Suppose the sequence 𝑆1 , 𝑆2 , … , 𝑆𝑛 has CDFs given by 𝐹𝑆 𝑛 𝑠 and the
random variable S has CDF 𝐹𝑆 𝑠 . Then 𝑆1 , 𝑆2 , … , 𝑆𝑛 converges to S in distribution if
lim 𝐹𝑆 𝑛 𝑠 = 𝐹𝑆 (𝑠)
𝑛→∞ 1 𝑠−𝜇
• Example: The same Sn = σ𝑖 𝑋𝑖 converges in distribution to 𝜇 since 𝐹𝑆 𝑛 𝑠 = 1 − Q and lim ቂ1 −
𝑠−𝜇 𝑛 𝜎/ 𝑛 𝑛→∞
Q 𝜎/ 𝑛 ቃ = 𝑢 𝑠 − 𝜇 = 𝐹𝑆 (𝑠)

46
Relations between convergence modes

47
Law of Large Numbers
• Weak LLN: Let 𝑋1 , 𝑋2 ,1… , 𝑋𝑛 be any n IID random variables. The sequence of
sample means Sn = σ𝑖 𝑋𝑖 converges in probability to the true mean of the
𝑛
underlying distribution. 2
𝑣𝑎𝑟 𝑆𝑛 𝜎 /𝑛
• Proof: lim Pr 𝑆𝑛 − 𝜇 > 𝜀 ≤ = =0
𝑛→∞ 𝜀2 𝜀2

• Strong LLN: Let 𝑋1 , 𝑋2 , …1 , 𝑋𝑛 be any n IID random variables. The sequence

of sample means Sn = σ𝑖 𝑋𝑖 converges almost everywhere to the true
𝑛
mean of the underlying distribution.

• Note: The distribution of a RV can be estimated by estimating the mean of

the indicator RV. So the LLN tells us that Monte Carlo simulation is well
founded.
48
Central Limit Theorem
• Let 𝑋1 , 𝑋2 , 2… , 𝑋𝑛 be a sequence of n IID random variables with mean 𝜇𝑋 and
variance 𝜎𝑋 . Let
1 𝑛 𝑋𝑖 −𝜇𝑋
𝑍𝑛 = σ𝑖=1
𝑛 𝜎𝑋

• Then, 𝑍𝑛 converges in distribution to a standard normal distribution.

• Proof: See textbook (Taylor approximation of the characteristic function

around 𝜔 = 0)

• Intuition: The sum of a sufficiently large number of IID random variables is

approximately Gaussian distributed.

49
Backup slides

50