Chapter 5.

1 Two Discrete Random Variables

• Consider transmission of two binary bits

where 𝐾𝑖 = Number of errors occurred after observing the ith bit.
• Initially Pr[E] = 0.2 and Pr[Ec] = 0.8
• Errors tend to occur in bursts
• Two random variables (RVs) are clearly
Sample space probability K1 K2
0.6 K1 K2 probability
E 12 (0.2)(0.6)=0.12
0.2 11 (0.2)(0.4)=0.08
0.4
start 01 (0.8)(0.1)=0.08
0.1 00 (0.8)(0.9)=0.72
0.8
EC
0.9

Need joint probability to describe occurrence of two RVs.

M Tummala & C W Therrien 2012 1
Chapter 5.1 Two Discrete Random Variables
5.5.1 The joint PMF

Joint PMF
𝑓𝐾1𝐾2 𝑘1 , 𝑘2 = Pr[𝐾1 = 𝑘1 , 𝐾2 = 𝑘2 ]
Three-dimensional plot

K1K2 probability
𝑓𝐾1 𝐾2 𝑘1 , 𝑘2
1, 2 0.12
0.72 𝑘2 1, 1 0.08
0, 1 0.08
0.08 0.12
2 0, 0 0.72
0.08
1

0
෍ ෍ 𝑓𝐾1 𝐾2 𝑘1 , 𝑘2 = 1
1 𝑘1 𝑘2

𝑘1
M Tummala & C W Therrien 2012 2
Chapter 5.1 Two Discrete Random Variables
5.1.1 The joint PMF

Marginal PMF
𝑓𝐾1 𝑘1 = ෍ 𝑓𝐾1𝐾2 [𝑘1 , 𝑘2 ] 𝑓𝐾2 𝑘2 = ෍ 𝑓𝐾1𝐾2 [𝑘1 , 𝑘2 ]
𝑘2 𝑘1

𝑓𝐾1 𝐾2 [𝑘1 , 𝑘2 ]

𝑓𝐾1 [𝑘1 ]
0.72 𝑘2

0.80 0.08 0.12

2
0.08
1
0

1
0.20 𝑘1
0

1 𝑘1
M Tummala & C W Therrien 2012 3
Chapter 5.1 Two Discrete Random Variables
5.1.2 Independent random variables

Independence for Random Variables

Two RVs are independent if 𝑓𝐾1 ,𝐾2 𝑘1 , 𝑘2 = 𝑓𝐾1 𝑘1 𝑓𝐾2 [𝑘2 ]
Marginal PMF Joint PMF
𝑓𝐾1 [𝑘1 ]
𝑓𝐾1 ,𝐾2 𝑘1 , 𝑘2 k2
0.8
0.720 0.080 0
0.2 k1
0 0.080 0.120

k1
0 1 k2
𝑓𝐾1 𝑘1 𝑓𝐾2 [𝑘2 ] 0.72 0.16 0.12
𝑓𝐾2 [𝑘2 ] 0.72
k1 0.8 0.576 0.128 0.096
0.16 0.2 0.144 0.032 0.024
0.12

0 1 2
k2 Are these RVs independent?
M Tummala & C W Therrien 2012 4
Chapter 5.1 Two Discrete Random Variables
5.1.3 Conditional PMFs for discrete random variables

Conditional PMF
Definition: 𝑓𝐾1 |𝐾2 𝑘1 𝑘2 = Pr[𝐾1 = 𝑘1 |𝐾2 = 𝑘2 ]
𝑓𝐾1𝐾2 [𝑘1 , 𝑘2 ]
𝑓𝐾1|𝐾2 𝑘1 𝑘2 =
𝑓𝐾2 [𝑘2 ]

𝑓𝐾1|𝐾2 𝑘1 0 𝑓𝐾1|𝐾2 𝑘1 1 𝑓𝐾1|𝐾2 𝑘1 2

1.0 1.0

0.5 0.5

k1 0 1 k1 k1
0 1 0 1

Relations
෍ 𝑓𝐾1|𝐾2 [𝑘1 |𝑘2 ] = 1, ෍ 𝑓𝐾1|𝐾2 [𝑘1 |𝑘2 ] = ?
𝑘1 𝑘2
M Tummala & C W Therrien 2012 5
Chapter 5.1 Two Discrete Random Variables
5.1.4 Bayes’ rule for discrete random variables

Bayes’ Rule for PMF’s

𝑓𝐾2 |𝐾1 𝑘2 𝑘1 𝑓𝐾1 𝑘1
𝑓𝐾1 |𝐾2 𝑘1 𝑘2 =
𝑓𝐾2 𝑘2
𝑓𝐾2 |𝐾1 𝑘2 𝑘1 𝑓𝐾1 𝑘1
=
σ𝑘1 𝑓𝐾1 𝐾2 [𝑘1 , 𝑘2 ]
𝑓𝐾2 |𝐾1 𝑘2 𝑘1 𝑓𝐾1 𝑘1
=
σ𝑘1 𝑓𝐾2 |𝐾1 𝑘2 𝑘1 𝑓𝐾1 [𝑘1 ]

It is useful when it is desired to “work backward” and infer the probability of

some fundamental random variable (but usually unobservable) from another
related random variable that can be measured or observed directly

M Tummala & C W Therrien 2012 6

Chapter 5.2 Two Continuous Random Variables
5.2.1 Joint distributions

Two Continuous Random Variables 𝑋1, 𝑋2

• Joint CDF
𝐹𝑋1,𝑋2 𝑥1 , 𝑥2 = Pr 𝑋1 ≤ 𝑥1 , 𝑋2 ≤ 𝑥2
𝑥1 𝑥2
= න න 𝑓𝑋1,𝑋2 𝑧1 , 𝑧2 𝑑𝑧2 𝑑𝑧1
−∞ −∞

• Joint PDF
𝜕 2 𝐹𝑋1,𝑋2 (𝑥1 , 𝑥2 )
𝑓𝑋1,𝑋2 𝑥1 , 𝑥2 =
𝜕𝑥1 𝜕𝑥2

M Tummala & C W Therrien 2012 7

Chapter 5.2 Two Continuous Random Variables
5.2.1 Joint distributions

Properties of the joint PDF

𝑓𝑋1,𝑋2 𝑥1 , 𝑥2 ≥ 0

∞ ∞
න න 𝑓𝑋1,𝑋2 𝑥1 , 𝑥2 𝑑𝑥2 𝑑𝑥1 = 1 (Unit volume)
−∞ −∞

𝑏1 𝑏2
Pr 𝑎1 < 𝑋1 ≤ 𝑏1 , 𝑎2 < 𝑋2 ≤ 𝑏2 = න න 𝑓𝑋1,𝑋2 𝑥1 , 𝑥2 𝑑𝑥2 𝑑𝑥1
𝑎1 𝑎2

Pr 𝑎1 < 𝑋1 ≤ 𝑎1 + Δ1 , 𝑎2 < 𝑋2 ≤ 𝑎2 + Δ2 ≈ 𝑓𝑋1,𝑋2 𝑎1 , 𝑎2 Δ1 Δ2

M Tummala & C W Therrien 2012 8

 Chapter 5.2 Two Continuous Random Variables
5.2.1 Joint distributions

Interpretation of the Joint PDF as Probability

𝑓𝑋1,𝑋2 𝑥1 , 𝑥2
Pr 𝑎1 < 𝑋1 ≤ 𝑏1 , 𝑎2 < 𝑋2 ≤ 𝑏2
𝑏1 𝑏2
= න න 𝑓𝑋1,𝑋2 𝑥1 , 𝑥2 𝑑𝑥2 𝑑𝑥1
𝑎1 𝑎2

a2 b2
x2

Δ1
a1

Δ2
b1

x1 Pr 𝑎1 < 𝑋1 ≤ 𝑎1 + Δ1 , 𝑎2 < 𝑋2 ≤ 𝑎2 + Δ2 ≈ 𝑓𝑋1,𝑋2 𝑎1 , 𝑎2 Δ1 Δ2

M Tummala & C W Therrien 2012 9

Chapter 5.2 Two Continuous Random Variables
5.2.2 Marginal PDFs: Projections of the joint density

Joint – Marginal PDF Relations

∞
𝑓𝑋1 𝑥1 = න 𝑓𝑋1 ,𝑋2 𝑥1 , 𝑥2 𝑑𝑥2
−∞
∞
𝑓𝑋2 𝑥2 = න 𝑓𝑋1 ,𝑋2 𝑥1 , 𝑥2 𝑑𝑥1
−∞

Note that:

∞ ∞
න 𝑓𝑋1 𝑥1 𝑑𝑥1 = න 𝑓𝑋2 𝑥2 𝑑𝑥2 = 1
−∞ −∞

M Tummala & C W Therrien 2012 10

Chapter 5.2 Two Continuous Random Variables
5.2.2 Marginal PDFs: Projections of the joint density

Marginal PDF as a Projection

𝑓𝑋1 (𝑥1 )
𝑓𝑋1,𝑋2 𝑥1 , 𝑥2

x2

∞
𝑓𝑋1 𝑥1 = න 𝑓𝑋1,𝑋2 𝑥1 , 𝑥2 𝑑𝑥2
−∞

X1 x1
M Tummala & C W Therrien 2012 11
Chapter 5.2 Two Continuous Random Variables
5.2.2 Marginal PDFs: Projections of the joint density
x1
Example
𝑐𝑒 −𝑥1 𝑒 −2𝑥2 , 0 ≤ 𝑥1 ≤ 𝑥2 < ∞
𝑓𝑋1,𝑋2 𝑥1 , 𝑥2 =ቊ
0, otherwise
x2

(a) Find the constant c :

∞ 𝑥2
1 = 𝑐 න න 𝑒 −𝑥1 𝑒 −2𝑥2 𝑑𝑥1 𝑑𝑥2
0 0
∞
= 𝑐න 1 − 𝑒 −𝑥2 𝑒 −2𝑥2 𝑑𝑥2
0
∞
𝑒 −2𝑥2 𝑒 −3𝑥2 𝑐
=𝑐 − =
−2 −3 0
6
𝑐=6
 c=6

M Tummala & C W Therrien 2012 12

Chapter 5.2 Two Continuous Random Variables
5.2.2 Marginal PDFs: Projections of the joint density
x1
Example (cont’d)
𝑐𝑒 −𝑥1 𝑒 −2𝑥2 , 0 ≤ 𝑥1 ≤ 𝑥2 < ∞
𝑓𝑋1,𝑋2 𝑥1 , 𝑥2 =ቊ
0, otherwise
x2

(b) Find 𝑓𝑋1 (𝑥1 )

∞
𝑓𝑋1 𝑥1 = 6 න 𝑒 −𝑥1 𝑒 −2𝑥2 𝑑𝑥2 = 3𝑒 −3𝑥1 , 0 ≤ 𝑥1 < ∞
𝑥1

(c) Find 𝑓𝑋2 (𝑥2 )

𝑥2
𝑓𝑋2 𝑥2 = 6 න 𝑒 −𝑥1 𝑒 −2𝑥2 𝑑𝑥2 = 6𝑒 −2𝑥2 1 − 𝑒 −𝑥2 , 0 ≤ 𝑥2 < ∞
0

M Tummala & C W Therrien 2012 13

Chapter 5.2 Two Continuous Random Variables
5.2.2 Marginal PDFs: Projections of the joint density

Properties of the joint CDF

1. 𝐹𝑋1,𝑋2 −∞, −∞ = 𝐹𝑋1,𝑋2 −∞, 𝑥2 = 𝐹𝑋1,𝑋2 𝑥1 , −∞ = 0
𝐹𝑋1,𝑋2 ∞, ∞ = 1
2. Marginal CDFs
𝐹𝑋1 𝑥1 = 𝐹𝑋1,𝑋2 𝑥1 , ∞ , 𝐹𝑋2 𝑥2 = 𝐹𝑋1,𝑋2 (∞, 𝑥2 )
3. If 𝑏1 > 𝑎1 and 𝑏2 > 𝑎2
𝐹𝑋1,𝑋2 𝑎1 , 𝑎1 ≥ 𝐹𝑋1,𝑋2 (𝑏1 , 𝑏2 )
4. Pr 𝑎1 < 𝑋1 ≤ 𝑏1 , 𝑎2 < 𝑋2 ≤ 𝑏2
= 𝐹𝑋1,𝑋2 𝑏1 , 𝑏2 − 𝐹𝑋1,𝑋2 𝑎1 , 𝑏2 − 𝐹𝑋1,𝑋2 𝑏1 , 𝑎2 + 𝐹𝑋1,𝑋2 𝑎1 , 𝑎2
x2
(a1,b2) (b1,b2)
b2

(a1,a2) (b1,a2)
a2

x1
M Tummala & C W Therrien 2012 a1 b1 14
Chapter 5.2 Two Continuous Random Variables
5.2.2 Marginal PDFs: Projections of the joint density

Uniform PDF Example x2

2

0.5, 0 ≤ 𝑥1 ≤ 1, 0 ≤ 𝑥2 < 2
𝑓𝑋1𝑋2 𝑥1 , 𝑥2 = ቊ
0, otherwise
x1
1

Find the joint CDF.

𝑥1 𝑥2
𝐹𝑋1𝑋2 𝑥1 , 𝑥2 = Pr 𝑋1 ≤ 𝑥1 , 𝑋2 ≤ 𝑥2 = න න 𝑓𝑋1,𝑋2 𝑧1 , 𝑧2 𝑑𝑧2 𝑑𝑧1
−∞ −∞

z2 z2
2 2
x2 x2
Some typical cases:
z1 z1
1 1
x1 x1
M Tummala & C W Therrien 2012 15
Chapter 5.2 Two Continuous Random Variables
5.2.2 Marginal PDFs: Projections of the joint density x2

Uniform PDF Example (cont’d) 2

𝑓𝑋1𝑋2 𝑥1 , 𝑥2
0.5, 0 ≤ 𝑥1 ≤ 1, 0 ≤ 𝑥2 < 2 x1
=ቊ 1
0, otherwise
Find the joint cdf.
𝑥1 𝑥2
𝐹𝑋1𝑋2 𝑥1 , 𝑥2 = Pr 𝑋1 ≤ 𝑥1 , 𝑋2 ≤ 𝑥2 = න න 𝑓𝑋1,𝑋2 𝑧1 , 𝑧2 𝑑𝑧2 𝑑𝑧1
−∞ −∞
z2 (v)
(iii)
2

(i) (ii) (iv)

(x1, x2)
z1
(i) 1 (i)

Case i x1 < 0 or x2 < 0 or both x1, x2 < 0

𝐹𝑋1,𝑋2 𝑥1 , 𝑥2 = 0

M Tummala & C W Therrien 2012 16

Chapter 5.2 Two Continuous Random Variables
5.2.2 Marginal PDFs: Projections of the joint density

Uniform PDF Example (cont’d)

Case ii 0 ≤ x1 ≤ 1, 0 ≤ x2 ≤ 2
1 𝑥1 𝑥2 1
𝐹𝑋1,𝑋2 𝑥1 , 𝑥2 = න න 𝑑𝑧2 𝑑𝑧1 = 𝑥1 𝑥2
2 0 0 2
z2 (v)
(iii)
Case iii 0 ≤ x1 ≤ 1, x2 > 2
2
1 𝑥1 2 (x1, x2)
𝐹𝑋1,𝑋2 𝑥1 , 𝑥2 = න න 𝑑𝑧2 𝑑𝑧1 = 𝑥1 (i) (ii) (iv)
2 0 0
(x1, x2) (x1, x2)
z1
Case iv x1 > 1, 0 ≤ x2 ≤ 2 (i) 1 (i)

1 1 𝑥2 1
𝐹𝑋1,𝑋2 𝑥1 , 𝑥2 = න න 𝑑𝑧2 𝑑𝑧1 = 𝑥2
2 0 0 2
Case v x1 > 1, x2 > 2
1 1 2
𝐹𝑋1,𝑋2 𝑥1 , 𝑥2 = න න 𝑑𝑧2 𝑑𝑧1 = 1
2 0 0
M Tummala & C W Therrien 2012 17
Chapter 5.2 Two Continuous Random Variables
5.2.3 Conditional PDFs: Slices of the joint density

Independent Random Variables

Two random variables X1 and X2 are said to be independent if

𝑓𝑋1,𝑋2 𝑥1 , 𝑥2 = 𝑓𝑋1 𝑥1 𝑓𝑋2 (𝑥2 ) for all 𝑥1 , 𝑥2

or (equivalently)

𝐹𝑋1,𝑋2 𝑥1 , 𝑥2 = 𝐹𝑋1 𝑥1 𝐹𝑋2 (𝑥2 ) for all 𝑥1 , 𝑥2

M Tummala & C W Therrien 2012 18

Chapter 5.2 Two Continuous Random Variables
5.2.3 Conditional PDFs: Slices of the joint density

Checking Independence – Example 1

6𝑒 −𝑥1 𝑒 −2𝑥2 , 0 ≤ 𝑥1 ≤ 𝑥2 < ∞

Given: 𝑓𝑋1𝑋2 𝑥1 , 𝑥2 =ቊ
0, 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
∞
find: 𝑓𝑋1 𝑥1 = 6 ‫ 𝑒 𝑥׬‬−𝑥1 𝑒 −2𝑥2 𝑑𝑥2 = 3𝑒 −3𝑥1 , 0 ≤ 𝑥1 < ∞
1

𝑥
and: 𝑓𝑋2 𝑥2 = 6 ‫׬‬0 2 𝑒 −𝑥1 𝑒 −2𝑥2 𝑑𝑥1 = 6𝑒 −2𝑥2 (1 − 𝑒 −𝑥2 ), 0 ≤ 𝑥2 < ∞

Are X1 and X2 independent?

check: 𝑓𝑋1,𝑋2 𝑥1 , 𝑥2 = 𝑓𝑋1 𝑥1 𝑓𝑋2 (𝑥2 )
6𝑒 −𝑥1 𝑒 −2𝑥2 ≠ 3𝑒 −3𝑥1 ⋅ 6𝑒 −2𝑥2 (1 − 𝑒 −𝑥2 ) ➔ not independent

M Tummala & C W Therrien 2012 19

Chapter 5.2 Two Continuous Random Variables
5.2.3 Conditional PDFs: Slices of the joint density

Checking Independence – Example 2

1 − 𝑒 −𝑎𝑥1 − 𝑒 −𝑏𝑥2 + 𝑒 − 𝑎𝑥1 +𝑏𝑥2 𝑥1 ≥ 0, 𝑥2 ≥ 0

𝐹𝑋1,𝑋2 𝑥1 , 𝑥2 =ቊ
0 otherwise
(a) Determine the marginal CDF:
𝐹𝑋1 𝑥1 = 𝐹𝑋1𝑋2 𝑥1 , ∞ = 1 − 𝑒 −𝑎𝑥1 𝑥1 ≥ 0

𝐹𝑋2 𝑥2 = 𝐹𝑋1𝑋2 ∞, 𝑥2 = 1 − 𝑒 −𝑏𝑥2 𝑥2 ≥ 0

(b) Obtain the marginal PDF from the CDF above:

𝑑𝐹𝑋1 (𝑥1 )
𝑓𝑋1 𝑥1 = = 𝑎𝑒 −𝑎𝑥1 𝑥1 ≥ 0
𝑑𝑥1

𝑑𝐹𝑋2 (𝑥2 )
𝑓𝑋2 𝑥2 = = 𝑏𝑒 −𝑎𝑥2 𝑥2 ≥ 0
𝑑𝑥2
M Tummala & C W Therrien 2012 20
Chapter 5.2 Two Continuous Random Variables
5.2.3 Conditional PDFs: Slices of the joint density

Checking Independence – Example 2 (cont’d)

(c) Check for independence using the CDF:

𝐶ℎ𝑒𝑐𝑘 𝐹𝑋1𝑋2 𝑥1 , 𝑥2 = 𝐹𝑋1 𝑥1 𝐹𝑋2 (𝑥2 )
1 − 𝑒 −𝑎𝑥1 − 𝑒 −𝑏𝑥2 + 𝑒 − 𝑎𝑥1 +𝑏𝑥2 = (1 − 𝑒 −𝑎𝑥1 )(1 − 𝑒 −𝑏𝑥2 )
Yes, they are independent

(d) Check for independence using the PDF:

𝜕𝐹𝑋1𝑋2 (𝑥1 , 𝑥2 )
𝑓𝑋1𝑋2 𝑥1 , 𝑥2 = = 𝑎𝑏𝑒 −𝑎𝑥1 𝑒 −𝑏𝑥2 𝑥1 ≥ 0, 𝑥2 ≥ 0
𝜕𝑥1 𝜕𝑥2
𝐶ℎ𝑒𝑐𝑘 𝑓𝑋1𝑋2 𝑥1 , 𝑥2 = 𝑓𝑋1 𝑥1 𝑓𝑋2 𝑥2
𝑎𝑏𝑒 −𝑎𝑥1 𝑒 −𝑏𝑥2 = 𝑎𝑒 −𝑎𝑥1 𝑏𝑒 −𝑏𝑥2
Yes, they are independent
M Tummala & C W Therrien 2012 21
Chapter 5.2 Two Continuous Random Variables
5.2.3 Conditional PDFs: Slices of the joint density

The Conditional PDF

Given two random variables X1 and X2, we define the conditional probability
density functions:
𝑓𝑋1 ,𝑋2 (𝑥1 , 𝑥2 )
𝑓𝑋1 |𝑋2 𝑥1 𝑥2 =
𝑓𝑋2 (𝑥2 )
𝑓𝑋1 ,𝑋2 (𝑥1 , 𝑥2 )
𝑓𝑋2 |𝑋1 𝑥2 𝑥1 =
𝑓𝑋1 (𝑥1 )

Note that:
∞ ∞
න 𝑓𝑋1 |𝑋2 𝑥1 𝑥2 𝑑𝑥1 = 1 while න 𝑓𝑋1 |𝑋2 𝑥1 𝑥2 𝑑𝑥2 =?
−∞ −∞

M Tummala & C W Therrien 2012 22

Chapter 5.2 Two Continuous Random Variables
5.2.3 Conditional PDFs: Slices of the joint density

Conditional PDF as a Slice through the Joint PDF

𝑓𝑋1 𝑋2 (𝑥1 , 𝑥2 )

slice at 𝑥20

𝑥20
x2

𝑓𝑋1,𝑋2 (𝑥1 , 𝑥20 )

𝑓𝑋1|𝑋2 𝑥1 𝑥20 =
X1 𝑓𝑋2 (𝑥20 )
x1
M Tummala & C W Therrien 2012 23
Chapter 5.2 Two Continuous Random Variables
5.2.3 Conditional PDFs: Slices of the joint density

Conditional PDF Example

6𝑒 −𝑥1 𝑒 −2𝑥2 , 0 ≤ 𝑥1 ≤ 𝑥2 < ∞
𝑓𝑋1𝑋2 𝑥1 , 𝑥2 =ቊ
0, 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒

Recall from previous example:

𝑓𝑋1 𝑥1 = 3𝑒 −3𝑥1 , 0 ≤ 𝑥1 < ∞
𝑓𝑋2 𝑥2 = 6𝑒 −2𝑥2 (1 − 𝑒 −𝑥2 ), 0 ≤ 𝑥2 < ∞

Find conditional PDF:

M Tummala & C W Therrien 2012 24

Chapter 5.2 Two Continuous Random Variables
5.2.4 Bayes’ rule for continues random variables

Bayes’ Rule for Probability Density Functions

• For any two random variables X1 and X2, we can write:

𝑓𝑋2|𝑋1 𝑥2 𝑥1 𝑓𝑋1 (𝑥1 )

𝑓𝑋1|𝑋2 𝑥1 𝑥2 =
𝑓𝑋 2 (𝑥2 )
• Now since
∞ ∞
𝑓𝑋2 𝑥2 = න 𝑓𝑋1𝑋2 𝑥1 , 𝑥2 𝑑𝑥1 = න 𝑓𝑋1|𝑋2 𝑥2 |𝑥1 𝑓𝑋1 (𝑥1 )𝑑𝑥1
−∞ −∞

• We have

𝑓𝑋2|𝑋1 𝑥2 𝑥1 𝑓𝑋1 (𝑥1 )

𝑓𝑋1|𝑋2 𝑥1 𝑥2 = ∞
‫׬‬−∞ 𝑓𝑋2|𝑋1 𝑥2 |𝑥1 𝑓𝑋1 (𝑥1 )𝑑𝑥1

M Tummala & C W Therrien 2012 25

Chapter 5.2 Two Continuous Random Variables
5.2.4 Bayes’ rule for continues random variables

Example
• Given a conditional PDF and the marginal PDFs, find the “inverse”
conditional PDF using Bayes’ Rule.
𝑒 −𝑥1
𝑓𝑋1|𝑋2 𝑥1 𝑥2 = , 0 ≤ 𝑥1 ≤ 𝑥2
1 − 𝑒 −𝑥2
𝑓𝑋1 𝑥1 = 3𝑒 −3𝑥1 , 0 ≤ 𝑥1 < ∞
𝑓𝑋2 𝑥2 = 6𝑒 −2𝑥2 1 − 𝑒 −𝑥2 , 0 ≤ 𝑥2 < ∞

• The inverse PDF is found as

𝑓𝑋1|𝑋2 𝑥1 𝑥2 𝑓𝑋2 (𝑥2 )

𝑓𝑋2|𝑋1 𝑥2 𝑥1 =
𝑓𝑋1 (𝑥1 )

6𝑒 −𝑥1 𝑒 −2𝑥2 (1 − 𝑒 −𝑥2 )

= −3𝑥 −𝑥
= 2𝑒 2𝑥1 𝑒 −2𝑥2 , 𝑥1 ≤ 𝑥2 < ∞
3𝑒 1 1 − 𝑒 2
M Tummala & C W Therrien 2012 26
Chapter 5.3 Expectation and Correlation
5.3.1 Correlation and covariance

Moments Computed from the Joint PDF

mean

variance

correlation

covariance

M Tummala & C W Therrien 2012 27

Chapter 5.3 Expectation and Correlation
5.3.1 Correlation and covariance
Example
2
𝑓𝑋1𝑋2 𝑥1 , 𝑥2 =ቊ10𝑥2 0 ≤ 𝑥2 ≤ 𝑥1 ,0 ≤ 𝑥 ≤ 1
0 otherwise

1 𝑥12
𝑚1 = න න 𝑥1 10𝑥2 𝑑𝑥2 𝑑𝑥1
0 0
1 𝑥12
𝑚2 = න න 𝑥2 10𝑥2 𝑑𝑥2 𝑑𝑥1
0 0

1 𝑥12
𝑟1,2 = 𝐸 𝑋1 𝑋2 = න න 𝑥1 𝑥2 10𝑥2 𝑑𝑥2 𝑑𝑥1
0 0

𝑐1,2 = 𝐸 𝑋1 𝑋2 − 𝑚1 𝑚2 =
M Tummala & C W Therrien 2012 28
Chapter 5.3 Expectation and Correlation
5.3.1 Correlation and covariance

Correlation/Covariance Relations
Relation
Cov 𝑋𝑖 , 𝑋𝑗 = 𝐸 𝑋𝑖 𝑋𝑗 − 𝐸 𝑋𝑖 𝐸[𝑋𝑗 ]
𝑐𝑖𝑗 = 𝑟𝑖𝑗 − 𝑚𝑖 𝑚𝑗

If Cov 𝑋𝑖 , 𝑋𝑗 = 0 then Xi and Xj are uncorrelated.

Note in this case 𝐸 𝑋𝑖 𝑋𝑗 = 𝐸 𝑋𝑖 𝐸[𝑋𝑗 ]

(If 𝐸 𝑋𝑖 𝑋𝑗 = 0 then Xi and Xj are said to be orthogonal.)

M Tummala & C W Therrien 2012 29

Chapter 5.3 Expectation and Correlation
5.3.1 Correlation and covariance

Independence Versus “Uncorrelated”

• Independent random variables are uncorrelated.

proof:

• Uncorrelated random variables are not necessarily independent!

M Tummala & C W Therrien 2012 30

Chapter 5.3 Expectation and Correlation
5.3.1 Correlation and covariance

Summary of Correlation Relations

• Correlation 𝑟 = 𝐸 𝑋1 𝑋2 , −∞ < 𝑟 < ∞
• Covariance 𝑐 = 𝐸[(𝑋1 − 𝑚1 )(𝑋2 − 𝑚2 )], −∞ < 𝑐 < ∞
• The correlation coefficient of X1 and X2 is defined as
Cov[𝑋1 , 𝑋2 ]
𝜌≜ , 1≤𝜌≤1
𝜎𝑋1 𝜎𝑋2

• X1 and X2 are independent if 𝑓𝑋1𝑋2 𝑥1 , 𝑥2 = 𝑓𝑋1 𝑥1 𝑓𝑋2 (𝑥2 )

• X1 and X2 are uncorrelated if 𝑐 = 0
• Independent → Uncorrelated
• X1 and X2 are orthogonal if 𝑟 = 0
• If 𝐸 𝑋1 or 𝐸 𝑋2 is zero, uncorrelated orthogonal
M Tummala & C W Therrien 2012 31
Chapter 5.3 Expectation and Correlation
5.3.2 Conditional expectation

• The expectation of 𝑋1 given 𝑋2 = 𝑥2 : the mean of the conditional density

∞
𝐸 𝑋1 𝑥2 = න 𝑥1 𝑓𝑋1|𝑋2 𝑥1 𝑥2 𝑑𝑥1
−∞

• If the conditioning variable is a random variable

∞
𝐸 𝑋1 𝑋2 = න 𝑥1 𝑓𝑋1|𝑋2 𝑥1 𝑋2 𝑑𝑥1 (random variable)
−∞

M Tummala & C W Therrien 2012 32

Chapter 5.4 Gaussian Random Variables

Joint Gaussian PDF

1
𝑓𝑋1 𝑋2 (𝑥1 , 𝑥2 ) x2
2

x2 x1
(m1, m2) contour
𝑓𝑋1𝑋2 𝑥1 , 𝑥2 = constant
x1

M Tummala & C W Therrien 2012 33

Chapter 5.4 Gaussian Random Variables

Joint Gaussian PDF (cont’d)

Marginal densities

Conditional density

where

M Tummala & C W Therrien 2012 34

Chapter 5.5 Multiple Random Variables
5.5.2 Sums of random variables

Moments of the Sum of Two Random Variables

Let X1, X2 be random variables with sum 𝑌 = 𝑋1 + 𝑋2
Mean of Y is
𝐸 𝑋1 + 𝑋2 = 𝐸 𝑋1 + 𝐸[𝑋2 ]
Variance of Y is
var 𝑌 = var 𝑋1 + var 𝑋2 + 2cov(𝑋1 , 𝑋2 )

Proof of variance:

M Tummala & C W Therrien 2012 35

Chapter 5.5 Multiple Random Variables
5.5.2 Sums of random variables

PDF for a Sum of Two Random Variables

x2
Find the PDF of Y where Y = X1 + X2

Yy
x1

if X1, X2 are independent

Sum of two independent random variables: The PDF of the sum is a

convolution of the component PDFs.
M Tummala & C W Therrien 2012 36
Chapter 5.5 Multiple Random Variables
5.5.2 Sums of random variables

Example 1 – Exponential Random Variables

• X1, and X2 are lifetimes of two light bulbs that are used sequentially.

• The lifetimes Xi are independent and identically distributed:

𝑓𝑋𝑖 𝑥𝑖 = 𝜆𝑒 −𝜆𝑥𝑖 , 𝑥𝑖 ≥ 0 (l is the “mean time to failure”)

• Find the PDF for the combined lifetime: Y = X1 + X2.

∞
The PDF of Y is given by 𝑓𝑌 𝑦 = ‫׬‬−∞ 𝑓𝑋1 𝑥1 𝑓𝑋2 𝑦 − 𝑥1 𝑑𝑥1

Substitute for the PDFs of X1 and X2:

or [ Erlang distribution ]

M Tummala & C W Therrien 2012 37

Chapter 5.5 Multiple Random Variables
5.5.2 Sums of random variables

Example 2 – Uniform Random Variables

Y is the sum of two independent random variables: Y = X1 + X2.
𝑓𝑋1 (𝑥1 ) is uniform [0, a] and 𝑓𝑋2 (𝑥2 ) is uniform [0, b] with 0 < a < b.

𝑓𝑋1 (𝑥1 ) 1/a 𝑓𝑋2 (𝑥2 )

1/b
x1 x2
a b

Find 𝑓𝑌(𝑦):
Case 1 y < 0

𝑓𝑋2 (𝑦 − 𝑢)
𝑓𝑋1 (𝑢)
1/a
1/b ⇒ 𝑓𝑌 𝑦 = 0
u
y-b y a

M Tummala & C W Therrien 2012 38

Chapter 5.5 Multiple Random Variables
5.5.2 Sums of random variables

Case 2 0 ≤ y < a

𝑓𝑋2 (𝑦 − 𝑢)
𝑓𝑋1 (𝑢)
1/a
1/b
u
y-b y a

Case 3 a ≤ y < b

𝑓𝑋2 (𝑦 − 𝑢) 𝑓𝑋1 (𝑢)

1/a

1/b u
y-b a y

M Tummala & C W Therrien 2012 39

Chapter 5.5 Multiple Random Variables
5.5.2 Sums of random variables

Case 4 b ≤ y ≤ a + b

𝑓𝑋1 (𝑢)
𝑓𝑋2 (𝑦 − 𝑢)
1/a
1/b
u
y-b a y

Case 5 y > a + b

𝑓𝑋1 (𝑢)
𝑓𝑋2 (𝑦 − 𝑢)
1/a
1/b
u
a y-b y
M Tummala & C W Therrien 2012 40
Chapter 5.5 Multiple Random Variables
5.5.2 Sums of random variables

Summary

fY(y)

1/b

y
a b a+b

M Tummala & C W Therrien 2012 41

Chapter 5.5 Multiple Random Variables
5.5.2 Sums of random variables

Sum of Multiple Random Variables

Let X1, X2, …, Xn be n random variables with sum Y:

𝑌 = 𝑋1 + 𝑋2 + ⋯ + 𝑋𝑛

Mean of Y:
𝑛

𝐸 𝑌 = ෍ 𝐸[𝑋𝑖 ]
𝑖=1

Variance of Y:
𝑛 𝑛 𝑖−1

Var 𝑌 = ෍ Var [𝑋𝑖 ] + 2 ෍ ෍ Cov 𝑋𝑖 , 𝑋𝑗

𝑖=1 𝑖=1 𝑗=1

M Tummala & C W Therrien 2012 42

Chapter 5.5 Multiple Random Variables
5.5.2 Sums of random variables

• If Xi and Xj (i  j), are uncorrelated (or independent), the covariance is zero

𝑛

Var 𝑌 = ෍ Var [𝑋𝑖 ]

𝑖=1

• Independent and identically-distributed (IID) random variables are

mutually independent and all have the same CDF and PDF.

• Consider that X1, X2, …, Xn are IID, with mean mX and variance X2.
𝐸 𝑌 = 𝑛𝑚𝑋
Var 𝑌 = 𝑛X2

M Tummala & C W Therrien 2012 43

Chapter 5.5 Multiple Random Variables
5.5.2 Sums of random variables

• The moment generating function is

𝑀𝑌 𝑠 = 𝐸 𝑒 𝑠𝑌 = 𝐸 𝑒 𝑠 𝑋1 +𝑋2 +⋯+𝑋𝑛

=
=
=

• The product of MGFs corresponds to the convolution of PDFs

𝑓𝑌 = 𝑓𝑋1 ∗ 𝑓𝑋2 ∗ ⋯ ∗ 𝑓𝑋𝑛

M Tummala & C W Therrien 2012 44

Chapter 5.5 Multiple Random Variables
5.5.2 Sums of random variables

Example – Gaussian Random Variables

Let X1, X2, …, Xn be independent Gaussian random variables
with mean mi and variance i2.
Let Y = X1 + X2 + … + Xn, then

The following correspondence exists for the Gaussian PDF and its
characteristic function.
𝑥𝑖 −𝑚𝑖 2
1 −
2𝜎𝑖2
1
𝑚𝑖 𝑠+ 𝜎𝑖2 𝑠 2
𝑓𝑋𝑖 𝑥𝑖 = 𝑒 𝑀𝑋𝑖 𝑠 = 𝑒 2
2𝜋𝜎𝑖

s = j

M Tummala & C W Therrien 2012 45

Chapter 5.5 Multiple Random Variables
5.5.2 Sums of random variables

Example – Gaussian Random Variables (Cont’d)

𝑀𝑌 𝑠 =

By inverting the transform:

2
𝑦−𝑚𝑦
1 −
2𝜎𝑌2
𝑓𝑌 𝑦 = 𝑒
2𝜋𝜎𝑌
with 𝑚𝑦 = 𝑚1 + ⋯ + 𝑚𝑛 , 𝜎𝑌2 = 𝜎12 + ⋯ + 𝜎𝑛2

The sum of independent Gaussian random

variables is Gaussian.

M Tummala & C W Therrien 2012 46

Chapter 5.5 Multiple Random Variables
5.5.2 Sums of random variables

Example – Bernoulli random variables

• 𝑋𝑖 : Bernoulli RVs with probability 𝑝 for 1
• Probability generating function (PGF)

• PGF of Y is

• Y is a binomial random variable

M Tummala & C W Therrien 2012 47