Outline Introduction Instructor Information Introduction to Random Vectors Statistical Distance Sampling

Multivariate Statistics: Introduction

Xing Wang

Sep, 2025

1 / 30
Outline Introduction Instructor Information Introduction to Random Vectors Statistical Distance Sampling

Introduction

Instructor Information

Introduction to Random Vectors

Statistical Distance

Sampling

2 / 30
Outline Introduction Instructor Information Introduction to Random Vectors Statistical Distance Sampling

Multivariate Statistics: Introduction

Xing Wang

Sep, 2025

3 / 30
Outline Introduction Instructor Information Introduction to Random Vectors Statistical Distance Sampling

Multivariate Statistical Analysis

• Instructor: Xing Wang

• Email: xwang@[Link]
• Class meetings: Tu, Th 3:25-5:00
• Office hours: By appointment
• Class website: TBD
• Credit: 60/3.0

4 / 30
Outline Introduction Instructor Information Introduction to Random Vectors Statistical Distance Sampling

COURSE SUMMARY
The aims of this course are
• To illustrate extensions of univariate statistical methodology
to multivariate data.
• To introduce students to some of the distinctive statistical
methodologies which arise only in multivariate data.
• To introduce students to some of the computational
techniques required for multivariate analysis available in
standard statistical packages.
Topics include: multivariate techniques and analyses, multivariate
analysis of variance, principal component analysis and factor
analysis, canonical correlation analysis, cluster analysis,
discrimination and classification.

The course uses the R software for statistical computing. Students

are expected to be familiar with the usage of the software.
5 / 30
Outline Introduction Instructor Information Introduction to Random Vectors Statistical Distance Sampling

HANDOUTS AND TEXTBOOKS

Handouts corresponding to each lecture will be available on the

class website before each class. The required textbook for this
course is:

Johnson, R.A. and Wichern, D.W., 2007. Applied Multivariate

Statistical Analysis (6th Edition). Prentice Hall, Upper Saddle
River, NJ.

Reading assignments will be made primary in this book.

6 / 30
Outline Introduction Instructor Information Introduction to Random Vectors Statistical Distance Sampling

METHOD OF STUDENT EVALUATION

The course grade will be based on

• Homework assignments (30%),
• Midterm exam (30%)
• Final exam (40%).

7 / 30
Outline Introduction Instructor Information Introduction to Random Vectors Statistical Distance Sampling

PREREQUISITES

Students are expected to have a background in undergraduate

linear algebra, probability, mathematical statistics, and linear
regression. Computer programming knowledge on R and/or
C/C++ is required.

8 / 30
Outline Introduction Instructor Information Introduction to Random Vectors Statistical Distance Sampling

Random Vectors/Matrices
• A random vector is a vector of random variables
󰀵 󰀶
X1
󰀹 .. 󰀺
X=󰀷 . 󰀸
Xp
• The mean or expectation of X is defined as
󰀵 󰀶
E(X1 )
󰀹 󰀺
E(X) = 󰀷 ... 󰀸
E(Xp )
• A random matrix is a matrix of random variables Z = (Zij ).
Its expectation is given by E(Z) = (E(Zij )).
• A constant vector a is a vector of constants; a constant
matrix A is a matrix of constants.
9 / 30
Outline Introduction Instructor Information Introduction to Random Vectors Statistical Distance Sampling

Basic Operations

In the following, a is a constant vector; A, B, C are constant

matrices; X, Y, Z are random matrices.
• E(a) = a, E(A) = A
• E(X + Y) = E(X) + E(Y)
• E(AX) = AE(X)
• E(AZB + C) = AE(Z)B + C

10 / 30
Outline Introduction Instructor Information Introduction to Random Vectors Statistical Distance Sampling

Covariance matrix
If X is a random vector, the covariance matrix of X is defined as
󰀵 󰀶
var(X1 ) cov(X1 , X2 ) · · · cov(X1 , Xp )
󰀹cov(X2 , X1 ) var(X2 ) · · · cov(X2 , Xp )󰀺
󰀹 󰀺
cov(X) = 󰀹 .. .. .. .. 󰀺
󰀷 . . . . 󰀸
cov(Xp , X1 ) cov(Xp , X2 ) · · · var(Xp )

An alternative form is
󰀅 󰀆
cov(X) = E (X − E(X))(X − E(X))T

󰀵󰀳 󰀴 󰀶
X1 − E(X1 )
󰀹󰁅 .. 󰁆 󰀺
= E 󰀷󰁃 . 󰁄 (X1 − E(X1 ) · · · Xp − E(Xp ))󰀸
Xp − E(Xp )

11 / 30
Outline Introduction Instructor Information Introduction to Random Vectors Statistical Distance Sampling

• If X1 , · · · , Xp are independent, the covariances are 0 and the

covariance matrix is diag(σ12 , · · · , σp2 ), where
var(Xi ) = σi2 , i = 1, · · · , p.
• Symmetry: cov(X) = [cov(X)]T
• cov(X + a) = cov(X) if a is a constant vector.
• cov(AX) = Acov(X)AT if A is a constant matrix.
• cov(X) is positive semi-definite.
• cov(X) = E(XXT ) − E(X)E(X)T

12 / 30
Outline Introduction Instructor Information Introduction to Random Vectors Statistical Distance Sampling

Correlation matrix

The correlation matrix of X is defined as

corr(X) = [corr(Xi , Xj )]
󰀵 󰀶
1 corr(X1 , X2 ) · · · corr(X1 , Xp )
󰀹corr(X2 , X1 ) 1 · · · corr(X2 , Xp )󰀺
󰀹 󰀺
=󰀹 .. .. .. .. 󰀺
󰀷 . . . . 󰀸
corr(Xp , X1 ) corr(Xp , X2 ) ··· 1
Denote cov(X) by Σ = (σij ). The correlation matrix and
covariance matrix is related by
√ √ √ √
Σ = diag( σ11 , · · · , σpp ) × corr(X) × diag( σ11 , · · · , σpp )

13 / 30
Outline Introduction Instructor Information Introduction to Random Vectors Statistical Distance Sampling

If Xp×1 and Yq×1 are random vectors,

cov(X, Y) = [cov(Xi , Yj )]
󰀵 󰀶
cov(X1 , Y1 ) cov(X1 , Y2 ) · · · cov(X1 , Yq )
󰀹cov(X2 , Y1 ) cov(X2 , Y2 ) · · · cov(X2 , Yq )󰀺
󰀹 󰀺
=󰀹 .. .. .. .. 󰀺
󰀷 . . . . 󰀸
cov(Xp , Y1 ) cov(Xp , Y2 ) · · · cov(Xp , Yq ) p×q
An alternative form is
󰀅 󰀆
cov(X) = E (X − E(X))(Y − E(Y))T
󰀵󰀳 󰀴 󰀶
X1 − E(X1 )
󰀹󰁅 .. 󰁆 󰀺
= E 󰀷󰁃 . 󰁄 (Y1 − E(Y1 ) · · · Yq − E(Yq ))󰀸
Xp − E(Xp )

14 / 30
Outline Introduction Instructor Information Introduction to Random Vectors Statistical Distance Sampling

• If A and B are constant matrices, then

cov(AX, BY) = Acov(X, Y)BT
󰀗 󰀘
X
• Let Z = . Then
Y
󰀗 󰀘
cov(X) cov(X, Y)
cov(Z) =
cov(Y, X) cov(Y)

• Let E(X) = µ and cov(X) = Σ and A be a constant matrix.

Then 󰀅 󰀆
E (X − µ)T A(X − µ) = tr(AΣ)
• E(XT AX) = tr(AΣ) + µT Aµ

15 / 30
Outline Introduction Instructor Information Introduction to Random Vectors Statistical Distance Sampling

Distance

A distance metric is a function that measures the ”distance”

between two objects. It must satisfy:
• Non-negativity: d(x, y) ≥ 0
• Identity: d(x, y) = 0 ⇐⇒ x = y
• Symmetry: d(x, y) = d(y, x)
• Triangle inequality: d(x, z) ≤ d(x, y) + d(y, z)
Distance metrics are fundamental in:
• Machine learning and data mining
• Statistics and probability theory
• Pattern recognition and computer vision

16 / 30
Outline Introduction Instructor Information Introduction to Random Vectors Statistical Distance Sampling

Euclidean Distance: Definition

For two points x = (x1 , x2 , . . . , xn ) and y = (y1 , y2 , . . . , yn ) in

n-dimensional space:

󰁳
deuclid (x, y) = (x1 − y1 )2 + (x2 − y2 )2 + · · · + (xn − yn )2

In vector notation:
󰁴
deuclid (x, y) = 󰀂x − y󰀂2 = (x − y)T (x − y)

17 / 30
Outline Introduction Instructor Information Introduction to Random Vectors Statistical Distance Sampling

Statistical Distance: Overview

• Statistical distances measure the difference between: Random

variables, Probability distributions, Statistical populations.
• Key examples include: Mahalanobis distance, Kullback-Leibler
divergence, Bhattacharyya distance, Hellinger distance
• These metrics incorporate statistical properties like: Means
and variances, Covariance structure, Probability distributions

18 / 30
Outline Introduction Instructor Information Introduction to Random Vectors Statistical Distance Sampling

Mahalanobis Distance

For two vectors x and y with covariance matrix Σ:

󰁴
dmaha (x, y) = (x − y)T Σ−1 (x − y)
Special case: Distance from a point to a distribution
󰁴
dmaha (x, µ) = (x − µ)T Σ−1 (x − µ)

The Mahalanobis distance accounts for:

• Different scales of variables
• Correlations between variables

19 / 30
Outline Introduction Instructor Information Introduction to Random Vectors Statistical Distance Sampling

Kullback-Leibler Divergence

Measures the difference between two probability distributions P

and Q:
󰀗 󰀘 󰁝
P (X) p(x)
DKL (P 󰀂Q) = EX∼P log = p(x) log dx
Q(X) q(x)
Properties:
• Not symmetric: DKL (P 󰀂Q) ∕= DKL (Q󰀂P )
• Non-negative: DKL (P 󰀂Q) ≥ 0
• Zero only when P = Q
• Not a true metric (fails symmetry)

20 / 30
Outline Introduction Instructor Information Introduction to Random Vectors Statistical Distance Sampling

The sample
Data: Xn×p a sample of size n from a p-variate population
󰀵 󰀶 󰀵 T󰀶
x11 x12 · · · x1p x1
󰀹 x21 x22 · · · x2p 󰀺 󰀹xT2 󰀺
󰀺 󰀹
󰀹 󰀺
X=󰀹 . .. .. .. 󰀺 = 󰀹 .. 󰀺
󰀷 .. . . . 󰀸 󰀷 . 󰀸
xn1 xn2 · · · xnp xTn

n points in p-dimensional space

Sample mean vector

n
1󰁛
x̄ = xj = [x̄1 x̄2 · · · x̄p ]T
n
j=1

1 󰁓n
where x̄k = n j=1 xjk , k = 1, · · · , p
21 / 30
Outline Introduction Instructor Information Introduction to Random Vectors Statistical Distance Sampling

Sample variance-covariance matrix

󰀵 󰀶
s11 s12 · · · s1p
󰀹s21 s22 · · · s2p 󰀺
󰀹 󰀺
Sn = 󰀹 . .. . . .. 󰀺
󰀷 .. . . . 󰀸
sp1 sp2 · · · spp
󰁓n
where sik = n1 j=1 (xji − x̄i )(xjk − x̄k ), i = 1, · · · , p,
k = 1, · · · , p

1 󰁓n
Sn = n j=1 (xj − x̄)(xj − x̄)T

22 / 30
Outline Introduction Instructor Information Introduction to Random Vectors Statistical Distance Sampling

󰀵 󰀶
x11 x12 · · · x1p
󰀹 x21 x22 · · · x2p 󰀺
󰀹 󰀺 󰀅 󰀆
X=󰀹 . .. .. .. 󰀺 = y1 y2 · · · yp
󰀷 .. . . . 󰀸
xn1 xn2 · · · xnp
p vectors in n-dimensional space

23 / 30
Outline Introduction Instructor Information Introduction to Random Vectors Statistical Distance Sampling

Random sampling of the sample mean vector and

covariance matrix

To study the sampling variability of x̄ and Sn , define the random

matrix: 󰀵 󰀶 󰀵 T󰀶
X11 X12 · · · X1p X1
󰀹 X21 X22 · · · X2p 󰀺 󰀹XT 󰀺
󰀹 󰀺 󰀹 2󰀺
X=󰀹 . .. .. .. 󰀺 = 󰀹 .. 󰀺
󰀷 .. . . . 󰀸 󰀷 . 󰀸
Xn1 Xn2 · · · Xnp XTn
Xjk : random variable ∀j, k
Xj : random vector ∀j
X: random matrix

24 / 30
Outline Introduction Instructor Information Introduction to Random Vectors Statistical Distance Sampling

Definition: X1 , X2 , · · · , Xn form a random sample if their joint

density function is given by the product

f (x1 ) × f (x2 ) × · · · × f (xn )

where
f (xj ) = f (xj1 , xj2 , · · · , xjp )
is the density function for the jth row vector.

Note:
1. For a random sample, the measurements of the p variables in
a ”single” trial, such as XTj = [Xj1 , Xj2 , · · · , Xjp ], will
usually be correlated. The measurements from ”different”
trials must be independent.
2. The independence of measurements from trial to trial may not
hold when variables are likely to drift over time (longitudinal
measurements).
25 / 30
Outline Introduction Instructor Information Introduction to Random Vectors Statistical Distance Sampling

Result: Let X1 , X2 , · · · , Xn be a random sample from a joint

distribution that has mean vector µ and covariance matrix Σ.

Then
1
E(X̄) = µ and cov(X̄) = Σ.
n
For the covariance matrix Sn ,
n−1
E(Sn ) = Σ.
n
n
So X̄ and n−1 Sn = S are unbiased estimators of µ and Σ,
respectively.

Proof: see Johnson and Wichern (2007) pages 121-122

26 / 30
Outline Introduction Instructor Information Introduction to Random Vectors Statistical Distance Sampling

Generalized variance

To provide one way of writing the information on all variances and

covariances in a covariance matrix as a single number

Definition: Generalized sample variance = |S| (i.e., the

determinant of S)
• See Johnson and Wichern (2007) pages 123-137 for details

27 / 30
Outline Introduction Instructor Information Introduction to Random Vectors Statistical Distance Sampling

Matrix operations of sample values

x̄k = ykT 1/n

󰀵 󰀶
x̄1
󰀹 .. 󰀺 1 T
x̄ = 󰀷 . 󰀸 = X 1
n
x̄p
󰀵 󰀶
x̄1 x̄2 · · · x̄p
1 󰀹 .󰀺
1x̄T = 11T X = 󰀷 ... .. . .
. . .. 󰀸
n
x̄1 x̄2 · · · x̄p
󰀵 󰀶
x11 − x̄1 · · · x1p − x̄p
1 󰀹 .. .. .. ..󰀺
X − 11T X = 󰀷 . . . .󰀸
n
xn1 − x̄1 · · · xnp − x̄p
1 1 1 1
(n−1)S = (X− 11T X)T (X− 11T X) = XT (I− 11T )T (I− 11T )X
n n n n
1
(n − 1)S = XT (I − 11T )X 28 / 30
Outline Introduction Instructor Information Introduction to Random Vectors Statistical Distance Sampling

1
Let D(p×p)
2
be the sample standard deviation
󰀵√ 󰀶
s11 0 ··· 0
󰀹 √
1 󰀹 0 s22 · · · 0 󰀺 󰀺
D2 = 󰀹 .. .. .. .. 󰀺
󰀷 . . . . 󰀸
√
0 0 ··· spp
1 1
D− 2 = (D 2 )−1

The sample correlation matrix

󰀵 s1p 󰀶
1 √ s12
√ ··· √ √
s11 s22 s11 spp
󰀹 √ s21√ 1 ···
s
√ 2p √
󰀺
1 1 󰀹 s22 s11 s22 spp 󰀺
R = D− 2 SD− 2 = 󰀹 󰀹 .. .. .. ..
󰀺
󰀺
󰀷 . . . . 󰀸
sp1 sp2
√ √
spp s11
√ √
spp s22 ··· 1

29 / 30
Outline Introduction Instructor Information Introduction to Random Vectors Statistical Distance Sampling

The linear combination of variables:

cT X = c 1 X 1 + · · · + c p X p

• Sample mean of cT X = cT x̄
• Sample variance of cT X = cT Sc
• Sample covariance of cT X and bT X = cT Sb

30 / 30