Lecture Note

Introductory Multivariate Methods

Stat 3133

By:-Berhanie A.
Awol S.
Department
DepartmentofofStatistics
Statistics
College of Computing
College & Informatics
of Science
Haramaya University
Dire Dawa, Ethiopia
Bahir Dar University
c 2015/2016
Bahir, Ethiopia
Contents

1 Review of Matrix Algebra 1

1.1 Definition of Matrix and Vector . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1.1 Matrix Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.2 Eigen Values and Eigen Vectors . . . . . . . . . . . . . . . . . . . . 4
1.2 Spectral Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.3 Singular Value Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . 8

2 Aspects of Multivariate Analysis 11

2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.1.1 Objectives of Multivariate Analysis . . . . . . . . . . . . . . . . . . 11
2.1.2 Organization of Multivariate Data . . . . . . . . . . . . . . . . . . . 12
2.2 Random Vectors and Matrices . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.3 Distance of Vectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.4 Linear Combinations of Random Vectors . . . . . . . . . . . . . . . . . . . 18
2.5 Expected Value of the Sample Mean and Covariance Matrix . . . . . . . . 20

3 The Multivariate Normal Distribution 21

3.1 The Multivariate Normal Density and Its Properties . . . . . . . . . . . . . 21
3.1.1 Principal Axis of the Multivariate Normal Density . . . . . . . . . . 23
3.1.2 Further Properties of the Multivariate Normal Density . . . . . . . 27
3.2 Sampling from the Multivariate Normal Distribution . . . . . . . . . . . . 27
3.2.1 The Multivariate Normal Likelihood . . . . . . . . . . . . . . . . . 27
3.2.2 The Sampling Distribution of X̄ and S . . . . . . . . . . . . . . . . 28
3.2.3 Large Sample Behaviour of X̄ and S . . . . . . . . . . . . . . . . . 29

4 Inference about a Mean Vector 30

4.1 The Plausibility of µ0 as a Value for a Normal Population Mean µ. . . . . 30
4.2 Confidence Region for the Mean Vector µ . . . . . . . . . . . . . . . . . . 32
4.3 Simultaneous Confidence Statements . . . . . . . . . . . . . . . . . . . . . 34
4.4 The Bonferroni Method of Multiple Comparisons . . . . . . . . . . . . . . 36
4.5 Likelihood-Ratio Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
4.6 Large Sample Inference about µ . . . . . . . . . . . . . . . . . . . . . . . . 38

i
Introductory Multivariate Methods - Stat 3133 E-mail: [Link]@[Link]

5 Comparison of Several Multivariate Means 39

5.1 Dependent Samples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
5.1.1 Paired Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
5.1.2 A Repeated Measures Design for Comparing Treatments . . . . . . 43
5.2 Independent Samples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
5.2.1 Comparing Mean Vectors from Two Populations . . . . . . . . . . . 46
5.2.2 Comparison of Several Multivariate Population Means . . . . . . . 49

ii
Chapter 1

Review of Matrix Algebra

The study of multivariate methods is greatly facilitated by the use of matrix algebra. This
chapter presents a review of basic concepts of matrix algebra which are essential to both
geometrical interpretations and algebraic explanations of subsequent multivariate statisti-
cal techniques.

1.1 Definition of Matrix and Vector

A rectangular array of numbers with, for instance, n rows and p columns is called a matrix
of dimension n × p. It is written as:
 
x11 x12 . . . x1p
 x21 x22 . . . x2p 
X =  ..
 
.. . . .. 
 . . . . 
xn1 xn2 . . . xnp

A vector is a matrix of n × 1 real numbers x1 , x2 , . . . , xn and it is written as:

 
x1
 x2 
x =  ..  or x0 = x1 , x2 , · · · , xn .
 

 . 
xn

A vector has both magnitude (length) and direction. The length of a vector x0 = (x1 , x2 , . . . , xn ),
is defined by q √
Lx = x21 + x22 + · · · + x2n = x0 x.

1
Introductory Multivariate Methods - Stat 3133 E-mail: [Link]@[Link]

The length of a vector can be expanded and contracted by multiplying with a constant a.
 
ax1
 ax2 
ax =  .. 
 
 . 
axn
Such multiplication of a vector x by a scalar a changes the length as
q √
Lax = a2 x21 + a2 x22 + · · · + a2 x2n = |a| x0 x.
When |a| > 1, vector x is expanded. When |a| < 1, vector x is contracted. When |a| = 1,
there is no change. If a < 0, the direction of vector x is changed.

Choosing a = L−1 x , we obtain the unit vector Lax , which has length 1 and lies in the
direction of x.
 
x1 p
Example 1.1. If n = 2, consider the vector x = . The length of x is Lx = x21 + x22 .
x2
Geometrically, the length of a vector in two dimensions can be viewed as the hypotenuse
of a right triangle.

1.1.1 Matrix Characteristics

• Rank: The rank of a matrix A is the maximum number of linearly independent rows
(columns).

– A set of k vectors x1 , x2 , · · · , xk is said to be linearly independent if a1 x1 +

k
X
a2 x 3 + · · · + ak x k = ai xi = 0 only if a1 = a2 = · · · = ak = 0. That is, if
i=1
every ai is zero, the x1 , x2 , · · · , xk (columns) are linearly independent. Linear
independence implies every vector can not be written as a linear combination
of the other vectors. Vectors of the same dimension that are not linearly inde-
pendent are said to be linearly dependent which means at least one vector can
be written as a linearcombination
 of the
 other vectors.
3 2
Example 1.2. x1 = , x2 =
4 1
a1 x1 + a2 x2 = 0 ⇒
3a1 + 2a2 = 0
4a1 + a2 = 0
holds only if a1 = a2 = 0. This confirms that x1 and x2 are linearly independent.
3 2
In other words, the columns of matrix A = are linearly independent.
4 1

2
Introductory Multivariate Methods - Stat 3133 E-mail: [Link]@[Link]

    
1 1 1
Example 1.3. x1 =  2  , x2 =  5  , x3 =  −1 
0 1 −1
a1 x1 + a2 x2 + a3 x3 = 0 ⇒

a1 + a2 + a3 = 0

2a1 + 5a2 − a3 = 0
a2 − a3 = 0
⇒ a1 + 2a2 = 0. ⇒ If a1 = a2 = 0, then a3 = 0. If a1 = 1, then a2 = a3 = 0.5.
Therefore, x1 , x2 and x3 are not linearly independent.
– The row and column rank of a matrix are equal.
∗ Rank (A) ≥ 0
∗ Rank (A) ≤ min(n, p)
∗ Rank (A) = Rank (A0 )
∗ Rank (A) = Rank (A0 A) = Rank (AA0 )
– Trace: The trace of a matrix is the sum of its diagonal elements: tr(A) =
Xk
aii .
i=1

∗ tr(A ± B) = tr(A) ± trB)

∗ tr(cA) = c tr(A)
∗ tr(An×p Bp×n ) = tr(BA)
∗ tr(An×p Bp×q Cq×n ) = tr(CAB) = tr(BCA)

• Determinant: Det (A) = |A|

– |aA| = an |A|
– |AB| = |BA| = |A||B|

• Inverse: A square matrix A is said to be non-singular if its rank is equal to the

number of rows (columns).

– If a k × k matrix A is non-singular, then there exist a unique k × k matrix B

such that AB = BA = Ik×k .
∗ The matrix B is called inverse of A denoted by A−1 .
∗ A−1 exists if and only if the determinant of A is non-zero. And hence,
|A−1 | = |A|−1 .

• Positive Definite Matrix: A symmetric matrix A is said to be positive definite if

the quadratic form Q(x) = x0 Ax > 0 for all x 6= 0 where x0 = (x1 , x2 , · · · , xn ).

3
Introductory Multivariate Methods - Stat 3133 E-mail: [Link]@[Link]

   
1 2 2
Example 1.4. A = , x=
2 4 −1
  
0
  1 2 2
Q(x) = x Ax = 2, −1 =0
2 4 −1
⇒ A is not positive definite.

– A symmetric matrix A is said to be positive semi-definite if x0 Ax ≥ 0 for all

x 6= 0.

1.1.2 Eigen Values and Eigen Vectors

Let A be a k × k matrix and I be a k × k identity matrix. The scalars λ1 , λ2 , · · · , λk
satisfying the polynomial equation:

|A − λI| = 0

are called the eigen values (characteristics roots) of matrix A. These eigen values are
unique unless two or more eigen values are equal.

• The equation |A − λI| = 0 as a function of λ is called characteristic equation.

• The eigen values of a symmetric matrix with real elements are real. λi ’s can be
complex numbers if the matrix is not symmetric.

• The eigen values of a positive definite matrix are all positive. If a k × k symmetric
matrix is positive semi-definite of rank r (r < k), then it has r positive and (k − r)
zero eigen values.

• The eigen values of a diagonal matrix are the diagonal elements themselves.

• The eigen values of an idempotent matrix A, that is, A = A2 are 1 and 0.

Associated with every eigen value λi of a square matrix A, there is an eigen vector xi
whose elements satisfy the homogenous system of equations:

(A − λi I)xi = 0 ⇔ Axi = λi xi

• If |A − λi I| = 0, there exist at least one non-trivial solution (xi 6= 0).

• The elements of the vector xi are determined only up to a scaled factor because the
system is homogenous, we get only relationship like x1i = 5x2i because the number
of unknowns is greater than the number of equations.

– Since the values of the eigen vectors are trivial, normalizing makes them unique,
that is, the eigen vectors have a unit length.

4
Introductory Multivariate Methods - Stat 3133 E-mail: [Link]@[Link]

– The normalized eigen vector, ei , of xi is:

1 xi
ei = xi = p 0
Lxi xi xi

∗ ||ei || = e0i ei = 1, for all i.

x0 xj
∗ e0i ej = p i0 p 0 = 0 for all i 6= j.
xi xi xj xj
– The normalized eigen vectors are chosen to satisfy e01 e1 = e02 e2 = · · · = e0k ek = 1
and be mutually perpendicular, e0i ej = 0, i 6= j.
 
1 2
Example 1.5. Find the eigen values and eigen vectors of A = .
3 2
   
1 2 1 0 1−λ 2
|A − λI| = 0 ⇒ −λ =0⇒ =0
3 2 0 1 3 2−λ

(1 − λ)(2 − λ) − 6 = 0 ⇒ λ2 − 3λ − 4 = 0
Thus, the eigen values of A are λ1 = 4 and λ2 = −1.

To find the corresponding eigen vectors:

Axi = λi xi , i = 1, 2

• For λ1 = 4,     
1 2 x11 x11
Ax1 = λ1 x1 ⇒ =4
3 2 x21 x21
⇒ x11 + 2x21 = 4x11
3
⇒ x21 = x11
2
 
2
Let x11 = 2 ⇒ x21 = 3. Thus, x = - - not unique.
3
 
2
The normalized eigen vector of x = is
3
  " #
1 1 2 √2
13
e1 = p 0 x1 = √ = √3
.
x1 x1 4+9 3 13

Note that e01 e1 = 1.

5
Introductory Multivariate Methods - Stat 3133 E-mail: [Link]@[Link]

• For λ2 = −1,     
1 2 x12 x12
Ax2 = λ2 x2 ⇒ =4
3 2 x22 x22
⇒ x12 + 2x22 = −x12
⇒ x22 = −x12
 
1
Let x12 = 1 ⇒ x22 = −1. Thus, x = - - not unique.
−1
 
1
The normalized eigen vector of x = is
−1
  " 1
#
1 1 1 √
2
e2 = p 0 x2 = √ = .
x2 x2 1+1 −1 − √12

Note that e02 e2 = 1. Also, e1 and e2 are orthogonal (perpendicular), that is, e01 e2 = 0.

Example 1.6. Find the eigen values and corresponding eigen vectors of the following two
matrices:  
  13 −4 2
1 −5
A = ⇒ λ1 = 6, λ2 = −4 and B =  −4 13 −2  ⇒ λ1 = 18, λ2 =
−5 1
2 −2 10
9, λ3 = 9

1.2 Spectral Decomposition

Any symmetric square matrix can be can be constructed from its eigen values and eigen
vectors.
Let A be a k × k symmetric matrix having k non-zero eigen values λ1 , λ2 , · · · , λk with
normalized eigen vectors e1 , e2 , · · · , ek . Then, the spectral decomposition of A is given
by:
X k
0 0 0
A = λ1 e1 e1 + λ2 e2 e2 + · · · + λk ek ek = λj ej e0j .
j=1
 
1 2
Example 1.7. A =
2 −2

1−λ 2
• The eigen values: |A − λI| = =0
2 −2 − λ
⇒ λ2 + λ − 6 = 0. Thus, the eigen values of A are λ1 = 2 and λ2 = −3.

• The eigen vectors are:

6
Introductory Multivariate Methods - Stat 3133 E-mail: [Link]@[Link]

– For λ1 = 2,
    
1 2 x11 x11
Ax1 = λ1 x1 ⇔ =2
2 −2 x21 x21
 
1 2
⇒ x21 = x
2 11
⇒ x1 =
1
!
√2
5
The normalized eigen vector corresponding to λ1 = 2 is e1 = √1
.
5

– For λ2 = −3,
    
1 2 x12 x12
Ax2 = λ2 x2 ⇒ =2
2 −2 x22 x22
 
1
⇒ x22 = −2x12 ⇒ x2 =
−2
!
√1
5
The normalized eigen vector corresponding to λ2 = −3 is e2 = .
− √25

Note that e01 e1 = e02 e2 = 1 and e01 e2 = e02 e1 = 0.

We need to show A = λ1 e1 e01 + λ2 e2 e02 .

! !
√2 √1
 
1 2 
√2 √1
 
√1

= 2 √15 −3 5 − √25
2 −2 5
5 5 − √25 5

The matrix is written as a function of eigen values and normalized eigen vectors.

In matrix form, the spectral decomposition of A is:

A = OΛO 0
 
λ1 0 · · · 0
 0 λ2 · · · 0 
where O = (e1 , e2 , · · · , ek ) and Λ = diag(λ1 , λ2 , · · · , λk ) =  .. .
 
.. . . .
 . . . .. 
0 0 · · · λk
Note here that O O = OO = Ik×k (O is orthogonal, O = O 0 ).
0 0 −1

In the above example,

" #
√2 √1
 
5 5 2 0
O = (e1 , e2 ) = √1 , Λ=
5
− √25 0 −3

⇒ A = OΛO 0 .

7
Introductory Multivariate Methods - Stat 3133 E-mail: [Link]@[Link]

Again, using spectral decomposition, for a positive definite matrix A

A−1 = OΛ−1 O 0
 
1 1 1
where O = (e1 , e2 , · · · , ek ) and Λ−1 = diag , ,··· , .
λ1 λ2 λk
k
X 1
⇒ A−1 = ej e0j .
λ
j=1 j

Also,
1 1
A 2 = OΛ 2 O 0
1
p p p 
where O = (e1 , e2 , · · · , ek ) and Λ = diag
2 λ1 , λ2 , · · · , λk .

k
1
X
λj ej e0j .
p
⇒ A2 =
j=1

 
  13 −4 2
1 1 2
Example 1.8. Find A−1 and A 2 . A = and A =  −4 13 −2 
2 1
2 −2 10

1.3 Singular Value Decomposition

Let A be an m × k matrix. Then there exist and m × m orthogonal matrix U (i.e.,
U U 0 = I) and a k × k orthogonal matrix V (i.e., V V 0 = I) such that A = U ΛV 0 where
Λ is an m × k matrix with (i, i) entry λi ≥ 0 for i = 1, 2, · · · , min(m, k) and the other
entries are zero.

• U = (e1 , e2 , · · · , emin(m,k) ) where ei (i = 1, 2, · · · , min(m, k)) is the normalized eigen

vector corresponding to λi of the matrix AA0 .

• V = (e∗1 , e∗2 , · · · , e∗min(m,k) ) where e∗i (i = 1, 2, · · · , min(m, k)) is the normalized eigen
vector corresponding to λ∗i of the matrix A0 A.
 √ 
λ1 √0 · · · 0
 0 λ2 · · · 0 
• Λ =  ..
 
.. . . .. 
 . . .
p .

0 0 ··· λmin(m,k)
√
Note that λi is the eigen value of matrix A where λi is the eigen value of A0 A or
AA0 .

8
Introductory Multivariate Methods - Stat 3133 E-mail: [Link]@[Link]

 
3 1 1
Example 1.9. A =
−1 3  1
  3 −1  
0 3 1 1  11 1
AA = 1 3 = and
−1 3 1 1 11
  1 1  
3 −1   10 0 2
3 1 1
A0 A =  1 3  =  0 10 4 
−1 3 1
1 1 2 4 2
• Eigen values and eigen vectors corresponding to AA0 .
11 − λ 1
|AA0 − λI| = 0 ⇒ = 0 ⇒ (11 − λ)2 − 1 = 0
1 11 − λ
λ2 − 22λ + 120 = 0 ⇒ λ = 12 or λ = 10.
The eigen values of AA√0 or A0 A are λ
√1 = 12 and λ2 = 10 which implies the eigen
values of A to be λ1 = 12 and λ2 = 10.
– Eigen vector corresponding to λ1 = 12,
    
0 11 1 x11 x11
AA x1 = λ1 x1 ⇒ = 12 ⇒ x21 = x11
1 11 x21 x21
" #
√1
 
1
Let x11 = 1 ⇒ x21 = 1 ⇒ x1 = ⇒ e1 = √12
1 2

– Eigen vector corresponding to λ2 = 10,

    
0 11 1 x12 x12
AA x2 = λ2 x2 ⇒ = 10 ⇒ x22 = −x12
1 11 x22 x22
" #
√1
 
1 2
Let x12 = 1 ⇒ x22 = −1 ⇒ x2 = ⇒ e2 =
−1 − √12

p p   √12 0 
" #
√1 √1
U = (e1 , e2 ) = 2 2
and Λ = diag λ1 , λ2 = √
√1 − √12 0 10
2

• Eigen values and eigen vectors corresponding to A0 A.

10 − λ 0 2
0
|A A − λI| = 0 ⇒ 0 10 − λ 4 =0
2 4 2−λ
10 − λ 4 0 4 0 10 − λ
(10 − λ) −0 +2 =0
4 2−λ 2 2−λ 2 4

λ2 − 12λ = 0 ⇒ λ = 12 or λ = 10 or λ = 0.

9
Introductory Multivariate Methods - Stat 3133 E-mail: [Link]@[Link]

– Eigen vector corresponding to λ1 = 12,

    
10 0 2 x11 x11
A0 Ax1 = λ1 x1 ⇒  0 10 4   x21  = 12  x21 
2 4 2 x31 x31

10x11 + 0x21 + 2x31 = 12x11

0x11 + 10x21 + 4x31 = 12x21
2x11 + 4x21 + 2x31 = 12x31
⇒ x21 = 2x11 and x31 = x11  
√1
 
1 6
Let x11 = 1 ⇒ x21 = 2 and x31 = 1 ⇒ x1 =  2  ⇒ e∗1 =  √2
 
6 
1 √1
6

– Eigen vector corresponding to λ2 = 10,

    
10 0 2 x12 x12
A0 Ax2 = λ2 x2 ⇒  0 10 4   x22  = 10  x22 
2 4 2 x32 x32

10x12 + 0x22 + 2x32 = 10x12

0x12 + 10x22 + 4x32 = 10x22
2x12 + 4x22 + 2x32 = 10x32
⇒ x32 = 0 and x22 = − 12 x12 .
√2
   
2 5
Let x12 = 2 ⇒ x22 = 1 ⇒ x1 =  −1  ⇒ e∗2 =  − √15 
0 0
 1 2

√ √
6 5
V = (e∗1 , e∗2 ) = √2 − √15 

6 
√1 0
6

" # √ " #
√1 √1 √1 √2 √1
 
0 2 2 12 √0 6 6 6 3 1 1
A = U ΛV = √1
=
2
− √12 0 10 √2
5
− √15 0 −1 3 1

10
Chapter 2

Aspects of Multivariate Analysis

2.1 Introduction
Multivariate statistical analysis is concerned with data collected with several dimensions
of the same individual (subject or experimental unit). Using multivariate analysis, the
variables can be examined simultaneously in order to access the key features of the process.
It enables us to

• explore the joint performance of the variables and

• determine the effect of each variable in the presence of the others.

As in the univariate case, it is assumed that a random sample of of the multi-component

observations has been collected from different individuals. The data consists of simulta-
neous measurements on many response variables. The common source of each individual
observation will generally lead to dependence or correlation among the dimension (compo-
nents). And this is the feature that distinguishes multivariate data and techniques from
their univariate counterparts.

2.1.1 Objectives of Multivariate Analysis

The objectives of scientific investigations to which multivariate methods most naturally
lend themselves include the following:

1. Data reduction or structural simplification. The phenomenon being studied is

represented as simply as possible without sacrificing valuable information. This will
make interpretation easier. Example: principal component analysis.

2. Sorting and grouping. Groups of ”similar” objects or variables are created, based
upon measured characteristics. Example: discriminant analysis.

3. Investigation of the dependence among variables. Studying the covariance

structure will help to determine the nature of the relationships among variables.

11
Introductory Multivariate Methods - Stat 3133 E-mail: [Link]@[Link]

In multivariate study, the interest is on the off-diagonals (covariances). Are all the
variables mutually independent or are one or more variables dependent on the others?
If so, how? Example: canonical correlation analysis.

4. Prediction. The relationship between variables can be determined for the purpose
of predicting the values of one or more variables on the basis of observations on
the other variables. Example: multivariate linear regression, multivariate analysis of
variance.

5. Hypothesis testing. Specific statistical hypotheses can be tested to validate as-

sumptions or to reinforce prior convictions.

2.1.2 Organization of Multivariate Data

Most multivariate data sets can be represented in a rectangular format, in which the
elements of each row correspond to the variable values of a particular unit in the data set
and the elements of the columns correspond to the values taken by a particular variable.
Suppose there are p ≥ 2 variables (characteristics) measured from n items. Let xij denote
the value of the j th variable on the ith item (i = 1, 2, · · · , n and j = 1, 2, · · · , p, n >> p).
Consequently, the data can be displayed as follows:

Variable 1 Variable 2 ··· Variable j ··· Variable p

Item 1 x11 x12 ··· xlj ··· xlp
Item 2 x21 x22 ··· x2j ··· x2p
.. .. .. ... .. .. ..
. . . . . .
Item i xi1 xi2 ··· xij ... xip
.. .. .. .. .. .. ..
. . . . . . .
Item n xn1 xn2 ··· xnj ··· xnp

This can be written as a rectangular array, matrix, X of n rows and p columns:

x11 x12 · · · x1j · · · xlp

 
 x21 x22 · · · x2j · · · x2p 
 . .. .. .. .. 
 . ...
 . . . . . 

X=
 xi1 xi2 . . . xij . . . xip 

 . .. .. .. .. .. 
 .. . . . . . 
xn1 xn2 · · · xnj · · · xnp n×p
A single multivariate observation is the collection of measurements on p different variables

12
Introductory Multivariate Methods - Stat 3133 E-mail: [Link]@[Link]

on the same item. Each row of X represents a multivariate observation.

x11 x12 · · · x1j · · · xlp x01
   
← 1st multivariate observation
 x21 x22 · · · x2j · · · x2p  
 . x02  ← 2nd multivariate observation
 . .. ... .. .. ..   .. 
 . . . . .   . 
  
X= =
x x . . . x . . . x x0i  ← ith multivariate observation

 i1 i2 ij ip  
 . .. .. .. .. ..  .. 
 ..

. . . . .   . 
xn1 xn2 · · · xnj · · · xnp x0 ← nth multivariate observation
n

Descriptive Statistics
A large data set is bulky, and its very mass poses a serious obstacle to any attempt to
visually extract pertinent information. Much of the information contained in the data can
be assessed by calculating certain summary numbers, known as descriptive statistics. For
example, the arithmetic average, or sample mean, is a descriptive statistic that provides
a measure of location-that is, a ”central value” for a set of numbers. And the average of
the squares of the distances of all of the numbers from the mean provides a measure of the
spread, or variation, in the numbers.
n
1X
• Sample mean: x̄j = xij ; j = 1, 2, · · · , p
n i=1
n
1X
• Sample variance: s2j = sjj = (xij − x̄j )2 ; j = 1, 2, · · · , p
n i=1
n
1X
• Sample covariance between Xj and Xk : sjk = (xij − x̄j )(xik − x̄k ); j, k =
n i=1
1, 2, · · · , p; j 6= k. Note sjk = skj for all j and k.
sjk
• Sample correlation coefficient between variable j and k: rjk = √ √ ; j, k =
sjj skk
1, 2, · · · , p. Note rjk = rkj and rjk = 1 if j = k.
Although, the sign of the sample correlation and sample covariance are the same, the
correlation is ordinarily easier to interpret as:
– its magnitude is bounded, that is, −1 ≤ rjk ≤ 1 for all j and k.
– it is unitless.
– it takes the variability into account.
But the major disadvantage of correlation is it does not measure non-linear associa-
tions.
The descriptive statistics for all the p variables in terms of vector and matrix operations
are:

13
Introductory Multivariate Methods - Stat 3133 E-mail: [Link]@[Link]

   
xi1 x̄1
n n
1X 1 X xi2   x̄2 
• Sample mean vector: x̄ = xi = = .
   
.. ..
n i=1 n i=1 
 
.   . 
xip x̄p p×1

n
1X
• Sample variance-covariance matrix: Sn = (x − x̄)(x − x̄)0 .
n i=1
 
s11 s12 · · · s1p
 s21 s22 · · · s2p 
⇒ Sn = 
 
.. .. . . .. 
 . . . . 
sp1 sp2 · · · spp p×p
Consequently, the sample standard deviation matrix is written as:
 √ 
s11 0 ··· 0
√
1
 0 s22 · · · 0 
V =  ..
 
2
.. . . .. 
 . . . . 
√
0 0 ··· spp p×p

1 1
• Sample correlation matrix: Rn = (V 2 )−1 Sn (V 2 )−1
   
r11 r12 · · · r1p 1 r12 · · · r1p
 r21
  r21 1 · · · r2p 
r22 · · · r2p   
⇒ Rn =  .. . = .

.. . . .. . . . 
 . . . ..   .. . . .. 
rp1 rp2 · · · rpp rp1 rp2 · · · 1 p×p

1 1
Note Sn = V 2 RV 2 . Note also that Sn and R are symmetric and positive definite.

Example 2.1. Find the sample mean vector, covariance and correlation matrices for the
following data matrix.  
4 1
 −1 3 
3 5

We find three people, and here is what we observe (with notation:

x11 = 4, x21 = −1, x31 = 3 and x12 = 1, x22 = 3, x32 = 5. The data array would the look
like:
   
x11 x12 4 1
X =  x21 x22  =  −1 3 
x31 x32 3 5

14
Introductory Multivariate Methods - Stat 3133 E-mail: [Link]@[Link]

2.2 Random Vectors and Matrices

A random vector (matrix) is a vector (matrix) whose elements are random variables. Let
Xj be the j th variable, then

• Mean(Xj ) = µj = E(Xj ); j = 1, 2, · · · , p

• Var(Xj ) = σj2 = E(Xj − µj )2 ; j = 1, 2, · · · , p

• Cov(Xj , Xk ) = σjk = E(Xj − µj )(Xk − µk ); j, k = 1, 2, · · · , p

σjk
• Cor(Xj , Xk ) = ρjk = √ √ ; j, k = 1, 2, · · · , p
σjj σkk

Let X be an n × p random vector, i.e, X = (X1 , X2 , · · · , Xp )0 . Then the mean vector is:
     
X1 E(X1 ) µ1
X2  E(X2 ) µ2 
E(X) = E  ..  =  ..  =  ..  = µ
     
 .   .  .
Xp E(Xp ) µp

The population variance-covariance matrix is:

  
X1 − µ1
X2 − µ2   
Σ = E(X − µ)(X − µ)0 = E  X − µ , X − µ , · · · , X − µ
  
..  1 1 2 2 p 2 
 .  
Xp−µp

 
(X1 − µ1 )2 (X1 − µ1 )(X2 − µ2 ) · · · (X1 − µ1 )(Xp − µp )
(X2 − µ2 )(X1 − µ1 ) (X2 − µ2 )2 ··· (X2 − µ2 )(Xp − µp )
⇒Σ=E
 
.. .. .. .. 
 . . . . 
(Xp − µp )(X1 − µ1 ) (Xp − µp )(X2 − µ2 ) . . . (Xp − µp )2
 
E(X1 − µ1 )2 E(X1 − µ1 )(X2 − µ2 ) . . . E(X1 − µ1 )(Xp − µp )
E(X2 − µ2 )(X1 − µ1 ) E(X2 − µ2 )2 ... E(X2 − µ2 )(Xp − µp )
=
 
.. .. ... .. 
 . . . 
E(Xp − µp )(X1 − µ1 ) E(Xp − µp )(X2 − µ2 ) . . . E(Xp − µp )2
 
σ11 σ12 · · · σ1p
 σ21 σ22 · · · σ2p 
Thus, Σ = 
 
.. .. . . .. 
 . . . . 
σp1 σp2 · · · σpp p×p

15
Introductory Multivariate Methods - Stat 3133 E-mail: [Link]@[Link]

Consequently, the population standard deviation matrix is written as:

 √ 
σ11 0 ··· 0
√
1  0
 σ22 · · · 0 
V 2 =  ..

.. . .. .. 
 . . . 
√
0 0 ··· σpp p×p

1 1
Also, the population correlation matrix is ρ = (V 2 )−1 Σ(V 2 )−1 , that is,
 
1 ρ12 · · · ρ1p
 ρ21 1 · · · ρ2p 
ρ =  ..
 
.. . . .. 
 . . . . 
ρp1 ρp2 · · · 1 p×p

1 1
Note: Σ = V 2 ρV 2 .

2.3 Distance of Vectors

Most multivariate techniques are based upon the simple concept of distance. If the point
P = (x1 , x2 ) is the point on the XY plane, then the Euclidean (straight line) distance from
P to the origin O = (0, 0) is given by the Pythagorean theorem. That is,
p q
dE (O, P ) = (x1 − 0) + (x2 − 0) = x21 + x22 .
2 2

All the points (x1 , x2 ) that lie a constant distance, say c, from the origin satisfying the
equation q
c = x21 + x22 ⇔ c2 = x21 + x22
is called equation of a circle with radius c.
The Euclidean distance between two points P = (x1 , x2 ) and Q = (y1 , y2 ) in the two
dimensional space is p
dE (P, Q) = (x1 − y1 )2 + (x2 − y2 )2 .
Similarly, the Euclidean distance between P = (x1 , x2 , · · · , xp ) and Q = (y1 , y2 , · · · , yp ) in
the p dimensional space is
q p
dE (P, Q) = (x1 − y1 )2 + (x2 − y2 )2 + · · · + (xp − yp )2 = (x − y)0 (x − y).

Straight line or Euclidean distance is unsatisfactory for most statistical purposes. This is
because each co-ordinate contributes equally to the calculation of Euclidean distance. This
suggests a statistical measure of distance.

16
Introductory Multivariate Methods - Stat 3133 E-mail: [Link]@[Link]

A statistical distance takes into account the variability as well as the correlation unlike
the Euclidean distance. Suppose X 0 = (X1 , X2 , · · · , Xp ) follows a p dimensional dis-
tribution with E(X) = µ and a variance-covariance matrix Cov(X) = Σ. Suppose
x̄ = (x̄1 , x̄2 , · · · , x̄p )0 is a vector of means based on an n × p observed data matrix. The
statistical distance between x̄ and µ is given by
p
dS (x̄, µ) = (x̄ − µ)0 Σ−1 ((x̄ − µ)).
v  −1  
u
u
u σ 11 σ 12 · · · σ 1p x̄ 1 − µ 1
 σ21 σ22 · · · σ2p  x̄2 − µ2 

u   
⇒ dS (x̄, µ) = u u 1 x̄ − µ 1 , x̄ 2 − µ 2 , · · · , x̄ p − µ p . .. . . .. . 
 .. .   .. 
  
t . .
σp1 σp2 · · · σpp x̄p − µp

• If Σ = I, the Euclidean and statistical distances are equal.

• If σij = 0 for i 6= j, the statistical distances is given by:

s
(x1 − µ1 )2 (x2 − µ2 )2 (xp − µp )2
dS (x̄, µ) = + + ··· + .
σ11 σ22 σpp

If one component has much larger variance than another, it will contribute less to the
squared distance. Two highly correlated variables will contribute less than two variables
that are nearly uncorrelated. Essentially, the use of the inverse of the covariance matrix
eliminates the effect of correlation and standardizes all of the variables.
     
x̄1 µ1 4 0
Example 2.2. Let x = ,µ= and Σ = . The variability in the x1 direc-
x̄2 µ2 0 1
tion is greater than that in the x2 direction as σ11 = 4 > σ22 = 1.
p
Euclidean distance: dE = (x1 − µ1 )2 + (x2 − µ2 )2 .
p
Statistical distance: dS = (x − µ)0 Σ−1 (x − µ).
s 1   
0 x1 − µ 1
⇒ dS = (x1 − µ1 , x2 − µ2 ) 4
0 1 x2 − µ 2
r
(x1 − µ1 )2 (x2 − µ2 )2
⇒ dS = + ⇒ equation of ellipse
4 1
All points that lie a constant distance, say c = 2, from the theoretical mean (µ1 , µ2 ) satisfy
the equation:
(x1 − µ1 )2 (x2 − µ2 )2
+ = c2 = 4.
4 1
At x1 = µ1 , (x2 − µ2 )2 = 4 ⇒ x2 − µ2 = ±2 ⇒ x2 = µ2 ± 2.

17
Introductory Multivariate Methods - Stat 3133 E-mail: [Link]@[Link]

At x2 = µ2 , (x1 − µ1 )2 = 16 ⇒ x1 − µ1 = ±4 ⇒ x1 = µ1 ± 4.

P lotof theellipse
The ellipse stretches in the x1 direction as compared to that in the x2 direction because
of the larger variance in x1 (the ellipse is parallel to the x1 ). Having the same variance in
both axes, the equation will be simply a circle.

2.4 Linear Combinations of Random Vectors

1. Univariate case:

• E(a1 X1 ) = a1 E(X1 ) = a1 µ1 , a1 ∈ R.
• Var(a1 X1 ) = a21 Var(X1 ) = a21 σ11 , a1 ∈ R.

2. Bivariate case:

• Cov(a1 X1 , a2 X2 ) = a1 a2 Cov(X1 , X2 ) = a1 a2 σ12 , a1 , a2 ∈ R.

 
X1
• Given X =
X2
 
X1
a1 X1 + a2 X2 = (a1 , a2 ) = a0 X
X2

– E(a0 X) = E(a1 X1 + a2 X2 ) = a1 E(X1 ) + a2 E(X2 ) = a1 µ1 + a2 µ2

 
µ
⇒ E(a X) = (a1 , a2 ) 1 = a0 µ
0
µ2

– Var(a0 X) = Var(a1 X1 +a2 X2 ) = Var(a1 X1 )+Var(a2 X2 )+Cov(a1 X1 , a2 X2 )

Var(a0 X) = a21 σ11 + a22 σ22 + a1 a2 σ12

  
σ11 σ12 a1
= (a1 , a2 )
σ12 σ22 a2
= a0 Σa

3. Multivariate case: If X a p-dimensional random vector and a ∈ Rp , then the linear

combination a0 X is a one-dimensional random variable. That is,
 
X1
 X2 
X =  .. 
 
 . 
Xp

18
Introductory Multivariate Methods - Stat 3133 E-mail: [Link]@[Link]

 
X1
 X2 
Linear combination: a1 X1 + a2 X2 + · · · + ap Xp = (a1 , a2 , · · · , ap )  ..  = a0 X.
 
 . 
Xp

• E(a0 X) = a0 E(X) = a0 µ
• Var(a0 X) 
= a0 Σa 
σ11 · · · σ1p
 .. .. 
Here Σ =  . . 
σp1 · · · σpp
4. Consider q linear combinations of p random variables.
p
X
Z1 = a11 X1 + a12 X2 + · · · + a1p Xp = a1j Xj = a1 X
j=1
p
X
Z2 = a21 X1 + a22 X2 + · · · + a2p Xp = a2j Xj = a2 X
j=1
..
.
p
X
Zq = aq1 X1 + aq2 X2 + · · · + aqp Xp = aqj Xj = aq X
j=1

In matrix form:
    
Z1 a11 a12 · · · a1p Z1
Z2  a21 a22 · · · a2p  Z2 
 ..  =  .. ..   ..  ⇔ Z = AX
    
.. . .
.  . . . .  . 
Zq aq1 aq2 · · · aqp Zq
• E(Z) = E(AX) = AE(X) = Aµ
• Cov(Z) = Cov(AX) = AΣA0
Example 2.3. Find the mean vector and covariance matrix for the linear combinations:
Z1 = X1 − X2 and
 Z2 = X1 + X2 .
1 −1 X1
Z= = AX
1 1 X2
    
1 −1 µ1 µ1 − µ2
• E(Z) = AE(X) = Aµ = =
1 1 µ2 µ1 + µ2
• Cov(Z) = ACov(Z)A0
     
1 −1 σ11 σ12 1 1 σ11 − 2σ12 + σ22 σ11 − σ22
⇒ Cov(Z) = =
1 1 σ12 σ22 −1 1 σ11 − σ22 σ11 + 2σ12 + σ22

19
Introductory Multivariate Methods - Stat 3133 E-mail: [Link]@[Link]

2.5 Expected Value of the Sample Mean and Covari-

ance Matrix
Let X be a random matrix given by:
   
X11 X12 · · · Xlp X10
 X21 X22 · · · X2p   X20 
X =  .. =
   
.. .. .. .. 
 . . . .   . 
Xn1 Xn2 · · · Xnp Xn0

If X1 , X2 , · · · , Xn is a random sample from some joint distribution with mean vector µ

and covariance matrix Σ, then
n n
1X 1X
• E(X̄) = E(Xi ) = µ = µ. Thus, X̄ is an unbiased estimator of the mean
n i=1 n i=1
vector µ.

• Cov(X̄) = E(X̄ − µ)(X̄ − µ)0 .

 
E(X̄1 − µ1 )2 E(X̄1 − µ1 )(X̄2 − µ2 ) · · · E(X̄1 − µ1 )(X̄p − µp )
E(X̄2 − µ2 )(X̄1 − µ1 ) E(X̄2 − µ2 )2 · · · E(X̄2 − µ2 )(X̄p − µp )
Cov(X̄) = 
 
.. .. ... .. 
 . . . 
E(X̄p − µp )(X̄1 − µ1 ) E(X̄p − µp )(X̄2 − µ2 ) ··· E(X̄p − µp )2
 
1 1 1
 n σ11 n σ12 · · · n σ1p 
1 1 1 
 σ
 21 n 22 σ · · · σ2p 
= n n 
 .. .. ..
.
.. 
 . . . 

1 1 1 
σp1 σp2 · · · σpp
n n n
1
= Σ
n
n
1X
Recall the sample variance-covariance matrix Sn = (xi − x̄)(xi − x̄)0 . It can
n i=1
n−1
be shown that E(Sn ) = Σ. Thus, Sn is a biased estimator of Σ. This implies
n
n
n 1 X
S = Sn = (xi − x̄)(xi − x̄)0 is an unbiased estimator of Σ, i.e.,
n−1 n − 1 i=1
n n n−1
E(S) = E(Sn ) = Σ = Σ.
n−1 n−1 n

20