Module-3

Multiple Random Variables

Naveen Kumar

Assistant Professor
SVNIT, Surat

October 13, 2025

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 1 / 86

Table of Contents

1 Introduction

2 Joint Distributions and Independence

3 Sums of Random Variables

4 Moment Generating Functions

5 Random Vectors
Normal (Gaussian) Random Vectors

6 Solved Problems

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 2 / 86

Table of Contents

1 Introduction

2 Joint Distributions and Independence

3 Sums of Random Variables

4 Moment Generating Functions

5 Random Vectors
Normal (Gaussian) Random Vectors

6 Solved Problems

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 3 / 86

Introduction

Concepts of one and two random variables can extend to more

variables.
Joint PDF, joint PMF, and joint CDF can be defined for three or
more random variables.
As the number of random variables increases, computations become
intractable.
Alternative techniques like moment generating functions and
probability bounds are typically used when dealing with multiple
random variables.

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 4 / 86

Table of Contents

1 Introduction

2 Joint Distributions and Independence

3 Sums of Random Variables

4 Moment Generating Functions

5 Random Vectors
Normal (Gaussian) Random Vectors

6 Solved Problems

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 5 / 86

Joint Distributions for Multiple Random Variables

Joint PMF for Discrete Random Variables:

Consider n discrete random variables X1 , X2 , . . . , Xn .
The joint probability mass function (PMF) is the probability that
each random variable takes a specific value:

PX1 ,X2 ,...,Xn (x1 , x2 , . . . , xn ) = P(X1 = x1 , X2 = x2 , . . . , Xn = xn )

Joint PDF for Continuous Random Variables:

For n jointly continuous random variables X1 , X2 , . . . , Xn , the joint
probability density function (PDF) defines the probability that the
variables fall within a set A ⊂ Rn .
Probability for a region A is given by integrating the joint PDF over A:
Z
P ((X1 , X2 , . . . , Xn ) ∈ A) = fX1 ,X2 ,...,Xn (x1 , x2 , . . . , xn ) dx1 dx2 . . . dxn
A

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 6 / 86

Marginal Distributions and Joint CDF

Marginal PDF:
The marginal distribution of one random variable X1 is obtained by
integrating out the other random variables from the joint PDF.
For example, the marginal PDF of X1 is:
Z ∞ Z ∞
fX1 (x1 ) = ··· fX1 ,X2 ,...,Xn (x1 , x2 , . . . , xn ) dx2 . . . dxn
−∞ −∞

Joint CDF for n Random Variables:

The joint cumulative distribution function (CDF) gives the
probability that each random variable is less than or equal to a given
value:

FX1 ,X2 ,...,Xn (x1 , x2 , . . . , xn ) = P(X1 ≤ x1 , X2 ≤ x2 , . . . , Xn ≤ xn )

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 7 / 86

Example: Finding the Constant c for the Joint PDF

Given: Let X , Y , Z be three jointly continuous random variables with the

joint PDF:
(
c(x + 2y + 3z), 0 ≤ x, y , z ≤ 1
fXYZ (x, y , z) =
0, otherwise

Find the constant c.

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 8 / 86

Example: Finding the Constant c for the Joint PDF
Given: Let X , Y , Z be three jointly continuous random variables with the joint PDF:
(
c(x + 2y + 3z), 0 ≤ x, y , z ≤ 1
fXYZ (x, y , z) =
0, otherwise

Find the constant c.

Solution: We know that the total probability must equal 1:
Z ∞ Z ∞ Z ∞
1= fXYZ (x, y , z) dx dy dz
−∞ −∞ −∞

This becomes: Z 1 Z 1 Z 1
1= c(x + 2y + 3z) dx dy dz
0 0 0

Simplifying:
Z 1 Z 1 
1
 Z 1 
3

= c + 2y + 3z dy dz = c + 3z dz
0 0 2 0 2
= 3c

Thus, c = 31 .

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 9 / 86

Example: Finding the Marginal PDF of X

Given: The joint PDF fXYZ (x, y , z) = 13 (x + 2y + 3z) for 0 ≤ x, y , z ≤ 1.

Find the marginal PDF of X .

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 10 / 86

Example: Finding the Marginal PDF of X

Given: The joint PDF fXYZ (x, y , z) = 13 (x + 2y + 3z) for 0 ≤ x, y , z ≤ 1.

Find the marginal PDF of X .
Solution: We find the marginal PDF fX (x) by integrating out y and z:
Z ∞ Z ∞ Z 1Z 1
1
fX (x) = fXYZ (x, y , z) dy dz = (x + 2y + 3z) dy dz
−∞ −∞ 0 0 3

Simplifying: Z 1
1 1 5
= (x + 1 + 3z) dz = (x + )
0 3 3 2
Thus, the marginal PDF of X is:
(
1 5


3 x+ 2 , 0≤x ≤1
fX (x) =
0, otherwise

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 11 / 86

Independence of Random Variables

Definition: Random variables X1 , X2 , . . . , Xn are independent if, for all

(x1 , x2 , . . . , xn ) ∈ Rn , the joint cumulative distribution function (CDF) is
the product of the individual CDFs:

FX1 ,X2 ,...,Xn (x1 , x2 , . . . , xn ) = FX1 (x1 )FX2 (x2 ) . . . FXn (xn )

For Discrete Random Variables: Random variables X1 , X2 , . . . , Xn are

independent if, for all (x1 , x2 , . . . , xn ) ∈ Rn , the joint probability mass
function (PMF) is the product of the individual PMFs:

PX1 ,X2 ,...,Xn (x1 , x2 , . . . , xn ) = PX1 (x1 )PX2 (x2 ) . . . PXn (xn )

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 12 / 86

Independence of Random Variables (Cont’d)

For Continuous Random Variables: Random variables X1 , X2 , . . . , Xn

are independent if the joint probability density function (PDF) is the
product of the individual PDFs:

fX1 ,X2 ,...,Xn (x1 , x2 , . . . , xn ) = fX1 (x1 )fX2 (x2 ) . . . fXn (xn )

Expectation for Independent Random Variables If random variables

X1 , X2 , . . . , Xn are independent, the expected value of their product is the
product of their expected values:

E [X1 X2 . . . Xn ] = E [X1 ]E [X2 ] . . . E [Xn ]

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 13 / 86

Independent and Identically Distributed (i.i.d.) Variables

Definition: Random variables X1 , X2 , . . . , Xn are said to be independent

and identically distributed (i.i.d.) if they are independent and have the
same marginal distributions:

FX1 (x) = FX2 (x) = · · · = FXn (x) for all x ∈ R

Expectation for i.i.d. Random Variables

If random variables X1 , X2 , . . . , Xn are i.i.d., then:

E [X1 X2 . . . Xn ] = E [X1 ]E [X2 ] . . . E [Xn ] = E [X1 ]n

- Because the Xi ’s are independent. - And the Xi ’s are identically

distributed, so E [X1 ] = E [X2 ] = · · · = E [Xn ].

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 14 / 86

Table of Contents

1 Introduction

2 Joint Distributions and Independence

3 Sums of Random Variables

4 Moment Generating Functions

5 Random Vectors
Normal (Gaussian) Random Vectors

6 Solved Problems

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 15 / 86

Sums of Random Variables

In many applications, we deal with the sum of several random variables.

Let Y be a random variable defined by the sum of n random variables:

Y = X1 + X2 + · · · + Xn

The Expectation of Y is given by the linearity of expectation:

E [Y ] = E [X1 ] + E [X2 ] + · · · + E [Xn ]

The Variance of a sum of two random variables X1 and X2 is:

Var(X1 + X2 ) = Var(X1 ) + Var(X2 ) + 2 · Cov(X1 , X2 )

- Variance: Measures the spread of a random variable.

- Covariance: Measures the relationship between two random variables.

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 16 / 86

Variance of the Sum of n Random Variables

For a random variable Y = X1 + X2 + · · · + Xn , the variance is:

 
n
X n
X Xn X n
Var(Y ) = Cov  Xi , Xj =
 Cov(Xi , Xj )
i=1 j=1 i=1 j=1

This simplifies to:

n
X X
Var(Y ) = Var(Xi ) + 2 Cov(Xi , Xj )
i=1 i<j

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 17 / 86

Variance for Independent Random Variables

If the random variables X1 , X2 , . . . , Xn are independent, then:

Cov(Xi , Xj ) = 0 for i ̸= j

Thus, the variance of the sum simplifies to:

n n
!
X X
Var Xi = Var(Xi )
i=1 i=1

Key takeaway: For independent random variables, the variance of their

sum is the sum of their individual variances.

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 18 / 86

Summary for Sums of Random Variables

- The sum of random variables is common in many applications. - The

expectation of the sum follows the linearity of expectation:

E [Y ] = E [X1 ] + E [X2 ] + · · · + E [Xn ]

- For dependent variables, the variance of the sum is:

n
X X
Var(Y ) = Var(Xi ) + 2 Cov(Xi , Xj )
i=1 i<j

- For independent random variables, the covariance terms vanish, and the
variance becomes the sum of individual variances:
n
X
Var(Y ) = Var(Xi )
i=1

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 19 / 86

Problem Overview

N people sit around a round table, where N > 5. Each person tosses a
coin. Anyone whose outcome is different from his/her two neighbors
receives a present. Let X be the number of people who receive presents.
Find E [X ] and Var(X ).

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 20 / 86

Indicator Random Variables

- Number the N people from 1 to N.

- Let Xi be the indicator random variable for the i-th person:

Xi = 1 if person i receives a present, and Xi = 0 otherwise.

- We can write the total number of presents as:

X = X1 + X2 + · · · + XN

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 21 / 86

Expectation of Xi and Total Expectation X

- First note that:

1
P(Xi = 1) =
4
This is because the probability that both neighbors have different
outcomes is:
1 1 1
P(Hi−1 , Ti , Hi+1 ) + P(Ti−1 , Hi , Ti+1 ) = + =
8 8 4
- Therefore, for each person: E [Xi ] = 14
- Using linearity of expectation, we can write:
1
E [X ] = E [X1 ] + E [X2 ] + · · · + E [XN ] = N ·
4
N
- Thus, the expected number of presents is: E [X ] = 4

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 22 / 86

Variance of X

- We can write the variance as:

N
X N X
X
Var(X ) = Var(Xi ) + Cov(Xi , Xj )
i=1 i=1 j̸=i

1


- Since Xi ∼ Bernoulli 4 , we have:

1 3 3
Var(Xi ) = · =
4 4 16

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 23 / 86

Finding Covariance Cov(X1 , X2 )
Xi and Xj are independent if there are at least two people between
them, i.e., 2 < |i − j| < N − 2. For these cases, Cov(Xi , Xj ) = 0.

Using symmetry:
Cov(X1 , X2 ) = Cov(X2 , X3 ) = · · · = Cov(XN , X1 )
We need to compute Cov(X1 , X2 ).
First, calculate E [X1 X2 ]:

E [X1 X2 ] = P(X1 = 1, X2 = 1) = P(HN , T1 , H2 , T3 )+P(TN , H1 , T2 , H3 )

1 1 1
= + =
16 16 8
Then:
Cov(X1 , X2 ) = E [X1 X2 ] − E [X1 ]E [X2 ] =
  
1 1 1 1
− =
8 4 4 16
Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 24 / 86
Finding Covariance Cov(X1 , X3 )

-Similarly Using symmetry:

Cov(X1 , X3 ) = Cov(X2 , X4 ) = · · · = Cov(XN , X2 )

- Now, calculate Cov(X1 , X3 ).

- First, calculate E [X1 X3 ]:

E [X1 X3 ] = P(X1 = 1, X3 = 1) = P(HN , T1 , H2 , T3 , H4 )

1 1 1
+P(TN , H1 , T2 , H3 , T4 ) = + =
32 32 16
- Then:
1 1
Cov(X1 , X3 ) = E [X1 X3 ] − E [X1 ]E [X3 ] = − =0
16 16

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 25 / 86

Finding Variance

- The variance of X can now be written as:

3
Var(X ) = N · + 2N · Cov(X1 , X2 ) + 2N · Cov(X1 , X3 )
16
- Substituting the values:
1 1 5N
Var(X ) = 3N · + 2N · =
16 16 16

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 26 / 86

PDF of Two Random Variables

We are interested in finding the probability density function (PDF) of

Y = X1 + X2 + · · · + Xn , where X1 , X2 , . . . , Xn are random variables.
For the case where Y = X1 + X2 and X1 and X2 are independent, the
PDF of Y is found by convolving the PDFs of X1 and X2 :
Z ∞
fY (y ) = fX1 (y ) ∗ fX2 (y ) = fX1 (x)fX2 (y − x)dx
−∞

If we extend this to the sum of more than two random variables, say
Y = X1 + X2 + · · · + Xn , we use the convolution formula repeatedly:

fY (y ) = fX1 (y ) ∗ fX2 (y ) ∗ · · · ∗ fXn (y )

However, this becomes computationally intensive as n increases.

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 27 / 86

Challenges of Convolution

Convolution works well for small numbers of random variables, but as

the number of variables increases, the complexity of computing the
convolution grows significantly.
It becomes necessary to look for alternative methods to simplify the
computation, especially for large n.

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 28 / 86

Table of Contents

1 Introduction

2 Joint Distributions and Independence

3 Sums of Random Variables

4 Moment Generating Functions

5 Random Vectors
Normal (Gaussian) Random Vectors

6 Solved Problems

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 29 / 86

Moment Generating Functions (MGFs)

One method that can simplify the process of finding the PDF of sums
of random variables is using moment generating functions
(MGFs).
They are particularly helpful because they provide information about
the moments of a random variable.
MGFs are used in deriving distributions of sums of independent
random variables.

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 30 / 86

Definition: Moments and Central Moments

Moments
The n-th moment of a random variable X is defined as:

E [X n ]

The n-th central moment of X is defined as:

E [(X − E [X ])n ]

Examples:
The first moment is the expected value E [X ].
The second central moment is the variance Var(X ), i.e.,
E [(X − E [X ])2 ].

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 31 / 86

Definition: Moment Generating Function (MGF)

The moment generating function (MGF) of a random variable X

is defined as:
MX (s) = E [e sX ]
The MGF MX (s) exists if there is a positive constant a such that
MX (s) is finite for all s ∈ [−a, a].
MGFs are useful because they encode all moments of a random
variable in a single function.

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 32 / 86

Example 1: Finding the MGF of Discrete Random Variable

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 33 / 86

Example 1: Finding the MGF of Discrete Random Variable

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 34 / 86

Why is the MGF Useful?

The moment generating function (MGF) of a random variable X is useful

for two main reasons:
Finding Moments: The MGF provides all the moments of the
random variable X , which is why it is called the moment generating
function.
Uniquely Determines the Distribution: If the MGF exists, it
uniquely determines the distribution of the random variable. This
means if two random variables have the same MGF, they must follow
the same distribution.
This makes the MGF particularly useful when analyzing sums of
independent random variables.

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 35 / 86

Finding Moments from the MGF

Let’s start by recalling the Taylor series expansion for e x , which is valid for
all x ∈ R:
∞
x2 x3 X xk
ex = 1 + x + + + ··· =
2! 3! k!
k=0

Similarly, we can express e sX as:

∞ ∞
X (sX )k X sk
e sX = = Xk
k! k!
k=0 k=0

Taking the expectation on both sides:

∞
X sk
MX (s) = E [e sX ] = E [X k ]
k!
k=0

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 36 / 86

Conclusion

From this expansion, we conclude that the k-th moment of X is the

k
coefficient of sk! in the Taylor series expansion of MX (s).

sk
E [X k ] = coefficient of in the Taylor series of MX (s)
k!
Therefore, if we have the Taylor series of the moment generating function
MX (s), we can extract all the moments of the random variable X .

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 37 / 86

Example

If Y ∼ Uniform(0, 1), find E [Y k ] using MY (s).

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 38 / 86

Solution

From Example 6.3, we know that the moment generating function (MGF)
of Y is:
es − 1
MY (s) = .
s
We can expand this expression as follows:
∞ ∞ ∞ k−1 ∞
!
1 X sk 1 X sk X s X 1 sk
MY (s) = −1 = = = .
s k! s k! k! k + 1 k!
k=0 k=1 k=1 k=0

sk 1
Thus, the coefficient of k! in the Taylor series of MY (s) is k+1 . Therefore,
we have:
1
E [Y k ] = .
k +1

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 39 / 86

Using Derivatives to Find Moments

k
From calculus, we know that the coefficient of sk! in the Taylor series of
MX (s) can also be obtained by taking the k-th derivative of MX (s) and
evaluating it at s = 0:

dk
E [X k ] = MX (s) .
ds k s=0

Thus, we can obtain all moments of X k from its MGF using this formula:
∞
X sk
MX (s) = E [X k ] ,
k!
k=0

and
dk
E [X k ] = MX (s) .
ds k s=0

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 40 / 86

Example: Exponential Distribution MGF and Moments

Let X ∼ Exponential(λ). Find the MGF MX (s) and all of its moments
E [X k ].

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 41 / 86

Finding the MGF

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 42 / 86

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 43 / 86
Example: Poisson Distribution MGF

Let X ∼ Poisson(λ). Find the MGF MX (s).

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 44 / 86

Finding the MGF

PMF:
e −λ λk
, for k = 0, 1, 2, . . .
PX (k) =
k!
The moment generating function (MGF) is given by:
∞
X e −λ λk
MX (s) = E [e sX ] = e sk
k!
k=0

Simplifying:
∞
X (λe s )k s
MX (s) = e −λ = e −λ e λe (Taylor series for e x )
k!
k=0

Therefore, the MGF is:

s
−1)
MX (s) = e λ(e , for all s ∈ R.

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 45 / 86

Theorem: Uniqueness of MGF

Consider two random variables X and Y . Suppose that there exists a

positive constant c such that the MGFs of X and Y are finite and
identical for all values of s in [−c, c]. Then,

FX (t) = FY (t), for all t ∈ R.

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 46 / 86

Example:

For a random variable X , we know that

2
MX (s) = , for s ∈ (−2, 2).
2−s
Find the distribution of X .
Solution:
We note that the above MGF is the MGF of an exponential random
variable with λ = 2 (Example 6.5). Thus, we conclude that
X ∼ Exponential(2).

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 47 / 86

Sum of Independent Random Variables

Suppose X1 , X2 , . . . , Xn are n independent random variables, and the

random variable Y is defined as:

Y = X1 + X2 + · · · + Xn .

Then,
h i
MY (s) = E [e sY ] = E e s(X1 +X2 +···+Xn ) = E [e sX1 e sX2 · · · e sXn ]

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 48 / 86

Sum of Independent Random Variables

MY (s) = E [e sX1 ]E [e sX2 ] · · · E [e sXn ] (since the Xi ’s are independent)

= MX1 (s)MX2 (s) · · · MXn (s).

Thus, if X1 , X2 , . . . , Xn are n independent random variables, then

MX1 +X2 +···+Xn (s) = MX1 (s)MX2 (s) · · · MXn (s).

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 49 / 86

Example: MGF of a Binomial Random Variable

If X ∼ Binomial(n, p), find the MGF of X .

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 50 / 86

Example: MGF of a Binomial Random Variable
If X ∼ Binomial(n, p), find the MGF of X .
Solution:
A Binomial random variable can be written as the sum of n independent
and identically distributed (i.i.d.) Bernoulli random variables:

X = X1 + X2 + · · · + Xn ,

where Xi ∼ Bernoulli(p). Thus,

MX (s) = MX1 (s)MX2 (s) · · · MXn (s) = (MX1 (s))n (since Xi ’s are i.i.d.).

We know that:
MX1 (s) = E [e sX1 ] = pe s + (1 − p).
Thus, the MGF of X is:

MX (s) = (pe s + 1 − p)n .

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 51 / 86

Table of Contents

1 Introduction

2 Joint Distributions and Independence

3 Sums of Random Variables

4 Moment Generating Functions

5 Random Vectors
Normal (Gaussian) Random Vectors

6 Solved Problems

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 52 / 86

Random Vectors

When dealing with multiple random variables, it is sometimes useful

to use vector and matrix notations.
This makes the formulas more compact and lets us use facts from
linear algebra.
When we have n random variables X1 , X2 , . . . , Xn , we can put them in
a (column) vector X:  
X1
X2 
X= . 
 
 .. 
Xn
We call X a random vector. Here, X is an n-dimensional vector.
Note: This book uses bold capital letters such as X, Y, and Z to represent
a random vector and bold lowercase letters x, y, and z are used to show
value of a random vector.
Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 53 / 86
 CDF of a Random Vector
We can write the cumulative distribution function (CDF) of the
random vector X as:

FX (x) = FX1 ,X2 ,...,Xn (x1 , x2 , . . . , xn ) = P(X1 ≤ x1 , X2 ≤ x2 , . . . , Xn ≤ xn )

PDF of a Random Vector

If the Xi ’s are jointly continuous, the probability density function
(PDF) of X can be written as:

fX (x) = fX1 ,X2 ,...,Xn (x1 , x2 , . . . , xn )

Expectation of a Random Vector

The expected value vector or the mean vector of the random vector X
is defined as:  
EX1
EX2 
EX =  . 
 
 .. 
EXn
Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 54 / 86
Random Matrices

A random matrix is a matrix whose elements are random variables. An

m × n random matrix M can be represented as:
 
X11 X12 . . . X1n
 X21 X22 . . . X2n 
M= .
 
. .. . . .. 
 . . . . 
Xm1 Xm2 . . . Xmn

We write this as M = [Xij ], where Xij is the element in the ith row and jth
column.

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 55 / 86

Mean Matrix and Linearity of Expectation for Random
Matrix

The mean matrix of M is given by:

 
EX11 EX12 ... EX1n
 EX21 EX22 ... EX2n 
EM =  .
 
.. .. .. 
 .. . . . 
EXm1 EXm2 . . . EXmn

Linearity of Expectation
The linearity of expectation holds for random vectors and matrices. For a
random vector X and Y = AX + b:

E Y = AE X + b

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 56 / 86

Correlation and Covariance Matrix

The correlation matrix RX of a random vector X is:

RX = E [XXT ]

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 57 / 86

Covariance Matrix
The covariance matrix CX is:
h i
CX = E (X − E X)(X − E X)T

Simplifies to
 
Var(X1 ) Cov(X2 , X1 ) . . . Cov(Xn , X1 )
Cov(X1 , X2 ) Var(X2 ) ... Cov(Xn , X2 )
CX = 
 
.. .. .. .. 
 . . . . 
Cov(X1 , Xn ) Cov(X2 , Xn ) . . . Var(Xn )
Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 58 / 86
Example

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 59 / 86

Example

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 60 / 86

Covariance Matrix

The covariance matrix is a generalization of the variance to random

vectors.
It is an important matrix and is used extensively. Concepts from linear
algebra, such as eigenvalues and positive definiteness, are often used
in its analysis.

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 61 / 86

Properties of the Covariance Matrix

Symmetry of the Covariance Matrix

For any random vector X, the covariance matrix CX is symmetric. If
CX = [cij ], then:

cij = Cov(Xi , Xj ) = Cov(Xj , Xi ) = cji

Thus, CX inherits all properties of symmetric matrices.

Properties of Symmetric Matrices:
CX can be diagonalized.
All eigenvalues of CX are real.
Assumption: X is a real random vector, meaning that the Xi ’s take
real values.

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 62 / 86

Positive Semi-Definiteness (PSD)

Definition: A symmetric matrix M is positive semi-definite (PSD) if, for

all vectors b:
bT Mb ≥ 0
Positive Definiteness (PD): M is positive definite if:

bT Mb > 0 for all b ̸= 0

Note: Every PD matrix is also PSD, but the converse is not true.

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 63 / 86

Theorem: Covariance Matrix is PSD

Theorem: Let X be a random vector with n elements. Then, its

covariance matrix CX is positive semi-definite (PSD).
Proof: Let b be any fixed vector with n elements. Define the random
variable Y as:
Y = bT (X − E X)
We have:
h i
0 ≤ E [Y 2 ] = E (YY T ) = bT E (X − E X)(X − E X)T b = bT CX b

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 64 / 86

Eigenvalues and Positive Definiteness

PSD Matrices: The eigenvalues of a PSD matrix are always ≥ 0.

PD Matrices: If all eigenvalues are strictly > 0, then the matrix is

positive definite (PD).

Theorem: CX is PD if and only if all eigenvalues are positive.

Equivalently, CX is PD if and only if:

det(CX ) > 0

Determinant and Eigenvalues

The determinant of a matrix is the product of its eigenvalues. Since all
eigenvalues of CX are ≥ 0, the determinant gives insight into whether CX
is PD or only PSD.

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 65 / 86

Example

Let X and Y be two independent Uniform(0,1) random variables. We

define the random vectors U and V as:
 
  X
X
U= , V= Y 
X +Y
X +Y

Our goal is to determine whether CU and CV are positive definite.

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 66 / 86

Finding CU is positive definite
We calculate CU as:
 
Var(X ) Cov(X + Y , X )
CU =
Cov(X , X + Y ) Var(X + Y )
Since X and Y are independent Uniform(0,1) random variables:
1 1
Var(X ) =
, Var(Y ) =
12 12
Variance and Covariance Calculations:
1
Var(X + Y ) = Var(X ) + Var(Y ) = 6
1 1
Cov(X , X + Y ) = Cov(X , X ) + Cov(X , Y ) = 12 +0= 12
Therefore,
1 1
 
CU = 12 12
1 1
12 6
Determinant of CU
1 1 1 1 1
det(CU ) =× − × = >0
12 6 12 12 144
Conclusion: CU is positive definite.
Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 67 / 86
Finding CV is Positive Definite

We calculate CV as:
 
Var(X ) Cov(Y , X ) Cov(X + Y , X )
CV =  Cov(X , Y ) Var(Y ) Cov(X + Y , Y )
Cov(X , X + Y ) Cov(Y , X + Y ) Var(X + Y )

Using independence and given variances:

1 1

12 0 12
1 1 
CV =  0 12 12
1 1 1
12 12 6

Determinant of CV
   
1 1 1 1 1 1 1 1
det(CV ) = × − × −0+ 0− × =0
12 12 6 12 12 12 12 12

Conclusion: CV is not positive definite but is positive semi-definite.

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 68 / 86

Definition of Jointly Normal Random Variables

Two random variables X and Y are bivariate normal or jointly

normal if aX + bY has a normal distribution for all a, b ∈ R.
This concept generalizes to n jointly normal random variables.
Random variables X1 , X2 , . . . , Xn are jointly normal if, for all
a1 , a2 , . . . , an ∈ R:

a1 X1 + a2 X2 + · · · + an Xn

is a normal random variable.

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 69 / 86

Gaussian Random Vector

 
X1
X2 
The normal (Gaussian) random vector X =  .  is a vector
 
 .. 
Xn
whose components X1 , X2 , . . . , Xn are jointly normal.
We denote this as X ∼ N(m, C), where:
   
E[X1 ] Var(X1 ) Cov(X1 , X2 ) ... Cov(X1 , Xn )
E[X2 ] Cov(X2 , X1 ) Var(X2 ) ... Cov(X2 , Xn )
m= .  and C= .
   
.. .. .. ..
 ..   . . . . 
E[Xn ] Cov(Xn , X1 ) ... ... Var(Xn )

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 70 / 86

Standard Normal Random Vector
Consider a standard normal vector Z with i.i.d. components Zi ∼ N(0, 1):
 
Z1
Z2 
Z =  . .
 
 .. 
Zn

The PDF of Z is:  

1 1 T
fZ (z) = exp − z z .
(2π)n/2 2
This is known as the standard normal random vector.
Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 71 / 86
General Normal Random Vector PDF

For a Gaussian random vector X ∼ N(m, C) with mean vector m and

covariance matrix C, the PDF is:
 
1 1 T −1
fX (x) = √ exp − (x − m) C (x − m) .
(2π)n/2 det C 2

Note: For derivation see book

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 72 / 86

Problem Statement

Let X and Y be two jointly normal random variables with:

X ∼ N(µX , σX ), Y ∼ N(µY , σY ),
ρ(X , Y ) = ρ. and
 
X
We aim to show that the general PDF formula for X = is equivalent
Y
to:
1
fX ,Y (x, y ) = p
2πσX σY 1 − ρ2
(x − µX )2 (y − µY )2
  
1 2ρ(x − µX )(y − µY )
exp − 2 + − .
2(1 − ρ2 ) σX σY2 σX σY

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 73 / 86

Solution Outline

To show equivalence, we will:

1 Identify the scaling factor a in both formulas.
2 Identify the exponent term b in both formulas.
 
X
We express X = with:
Y
     2 
µ Var(X ) Cov(X , Y ) σX ρσX σY
m= X and C= = .
µY Cov(Y , X ) Var(Y ) ρσX σY σY2

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 74 / 86

Determinant of Covariance Matrix C We compute det(C) as follows:

det(C) = σX2 σY2 (1 − ρ2 ).

Therefore, in the general formula for the PDF, the scaling factor a is:
1
a= p .
2πσX σY 1 − ρ2

This matches the factor a in the target formula.

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 75 / 86

Inverse of Covariance Matrix C−1
The inverse of C is calculated as:
σY2
 
−1 1 −ρσX σY
C = 2 2 .
σX σY (1 − ρ2 ) −ρσX σY σX2

Calculating the Exponent Term b

To find b, we compute:

(x − m)T C−1 (x − m),

   
x µ
where x = and m = X .
y µY

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 76 / 86

Detailed Computation of the Exponent Term Expanding the exponent
term:
T 
σY2
  
T −1 1 x − µX −ρσX σY x − µX
(x−m) C (x−m) = 2 2 .
σX σY (1 − ρ2 ) y − µY −ρσX σY σX2 y − µY

Final Form of the Exponent Term After matrix multiplication, we get:

(x − µX )2 (y − µY )2
 
T −1 1 2ρ(x − µX )(y − µY )
(x−m) C (x−m) = 2 + 2 − .
1 − ρ2 σX σY σX σY
 
X
Thus, the general PDF formula for X = matches the target formula
Y

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 77 / 86

Table of Contents

1 Introduction

2 Joint Distributions and Independence

3 Sums of Random Variables

4 Moment Generating Functions

5 Random Vectors
Normal (Gaussian) Random Vectors

6 Solved Problems

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 78 / 86

Problem 1: Finding the Joint PDF of X and Y

Problem: Let X , Y , and Z be three jointly continuous random variables

with joint PDF:
(
1
(x + 2y + 3z), 0 ≤ x, y , z ≤ 1
fXYZ (x, y , z) = 3
0, otherwise

Find the joint PDF of X and Y , fXY (x, y ).

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 79 / 86

Solution Problem 1

Solution:
Z 1  
1 1 3
fXY (x, y ) = (x + 2y + 3z) dz = x + 2y + , 0 ≤ x, y ≤ 1.
0 3 3 2

Thus,
(
1 3


3 x + 2y + 2 , 0 ≤ x ≤ 1, 0 ≤ y ≤ 1
fXY (x, y ) =
0, otherwise

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 80 / 86

Problem 2

Problem: Let X , Y , and Z be three independent random variables with

X ∼ N(µ, σ 2 ) and Y , Z ∼ Uniform(0, 2). Also, we know that

E [X 2 Y + XYZ ] = 13, E [XY 2 + ZX 2 ] = 14.

Find µ and σ.

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 81 / 86

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 82 / 86
Problem 4

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 83 / 86

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 84 / 86
Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 85 / 86
 Thanks

Naveen Kumar (SVNIT, Surat) MFCS October 13, 2025 86 / 86