Matrix Algebra and Random Vectors

Matrix Algebra and Random Vectors

Matrix Algebra and Random Vectors

Hanoi, 2022
Content

Matrix Algebra and Random Vectors 5

5.1 Revision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
5.1.1 Some basics of matrix and vector algebra . . . . . . . . . . . . 6
5.1.2 Random vectors and matrices . . . . . . . . . . . . . . . . . . 12
5.1.3 Mean Vectors and Covariance Matrices . . . . . . . . . . . . . 13
5.1.4 The Mean Vector and Covariance Matrix for Linear Combi-
nations of Random Variables . . . . . . . . . . . . . . . . . . . 19
5.2 Linear Algebra - linalg in Python . . . . . . . . . . . . . . . . . . . . 25
5.2.1 Matrix Operators . . . . . . . . . . . . . . . . . . . . . . . . . 25
5.3 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

3
Applied Multivariate Statistical Analysis

Matrix Algebra and Random Vectors

5.1. Revision 6

5.1 Revision

5.1.1 Some basics of matrix and vector algebra

Dependent and independent vectors

A pair of vectors x and y of the same dimension is said to be linearly dependent if

there exist constants c1 and c2 , both not zero, such that

c1 x + c2 y = 0

A set of vectors x1 , x2 , . . . , xk is said to be linearly dependent if there exist constants

c1 , c2 , . . . , ck , not all zero, such that

c1 x 1 + c2 x 2 + · · · + ck x k = 0

Linear dependence implies that at least one vector in the set can be written as a
linear combination of the other vectors. Vectors of the same dimension that are not
linearly dependent are said to be linearly independent.

The projection of a vector

The projection (or shadow) of a vector x on a vector y is

(x′ y) (x′ y) 1
Projection of x on y = y′ y
y = Ly Ly
y,

where the vector L−1

y y has unit length.

The length of the projection is

|x′ y| x′ y
Length of protection = = Lx = Lx | cos(θ)|,
Ly Lx Ly

where θ is the angle between x and y. (See Figure 2.5.)

Nguyễn Thị Nhung - Đại học Thăng Long

5.1. Revision 7

The orthogonal matrices

Another special class of square matrices with which we shall become familiar are
the orthogonal matrices, characterized by

QQ′ = Q′ Q = I or Q′ = Q−1

The name derives from the property that if Q has i th row q′i , then QQ′ = I implies
that q′i qi = 1 and q′i qj = 0 for i ̸= j, so the rows have unit length and are mutually
perpendicular (orthogonal). According to the condition Q′ Q = I, the columns have
the same property.
We conclude our brief introduction to the elements of matrix algebra by intro-
ducing a concept fundamental to multivariate statistical analysis. A square matrix
A is said to have an eigenvalue λ, with corresponding eigenvector x ̸= 0, if

Ax = λx

Ordinarily, we normalize x so that it has length unity; that is, 1 = x′ x. It is conve-

nient to denote normalized eigenvectors by e, and we do so in what follows. Sparing
you the details of the derivation (see [1], we state the following basic result:
Let A be a k × k square symmetric matrix. Then A has k pairs of eigenvalues
and eigenvectors namely,
λ1 , e1 λ2 , e2 . . . λk , ek
The eigenvectors can be chosen to satisfy 1 = e′1 e1 = · · · = e′k ek and be mutually
perpendicular. The eigenvectors are unique unless two or more eigenvalues are equal.

Nguyễn Thị Nhung - Đại học Thăng Long

5.1. Revision 8

POSITIVE DEFINITE MATRICES

The study of the variation and interrelationships in multivariate data is often based
upon distances and the assumption that the data are multivariate normally dis-
tributed. Squared distances (see Chapter 1) and the multivariate normal density
can be expressed in terms of matrix products called quadratic forms (see Chapter
4). Consequently, it should not be surprising that quadratic forms play a central
role in multivariate analysis. In this section, we consider quadratic forms that are
always nonnegative and the associated positive definite matrices.
Results involving quadratic forms and symmetric matrices are, in many cases, a
direct consequence of an expansion for symmetric matrices known as the spectral
decomposition. The spectral decomposition of a k × k symmetric matrix A is given
by 1 where λ1 , λ2 , . . . , λk are the eigenvalues of A and e1 , e2 , . . . , ek are the associated
normalized eigenvectors. (See also Result 2A.14 in Supplement 2A). Thus, e′i ei = 1
for i = 1, 2, . . . , k, and e′i ej = 0 for i ̸= j.

Example 5.1.1 (The spectral decomposition of a matrix). Consider the symmetric

matrix  
13 −4 2
A =  −4 13 −2 
 

2 −2 10
The eigenvalues obtained from the characteristic equation |A − λI| = 0 are λ1 =
9, λ2 = 9, and λ3 = 18 (Definition 2A.30). The corresponding eigenvectors e1 , e2 ,
and e3 are the (normalized) solutions of the equations Aei = λi ei for i = 1, 2, 3.
Thus, Ae1 = λe1 gives
    
13 −4 2 e11 e11
 −4 13 −2   e21  = 9  e21 
    

2 −2 10 e31 e31
or
13e11 − 4e21 + 2e31 = 9e11
−4e11 + 13e21 − 2e31 = 9e21
2e11 − 2e21 + 10e31 = 9e31

The spectral decomposition of A is then

A = λ1 e1 e′1 + λ2 e2 e′2 + λ3 e3 e′3

Nguyễn Thị Nhung - Đại học Thăng Long

5.1. Revision 9

or
1
 
  √
13 −4 2  2  
1  √1 1
 
 −4 13 −2  = 9 
 
√ √ 0

2
 2 2
2 −2 10  
0
1
 
2
 
√
 18   3 
−1 −1 −4
 
 √1 2  2 2 1
  
+9
 √  √ √ + 18  −  3 −3

 18  18 18 18  3 3
 −4   1 
√
18 3

The spectral decomposition is an important analytical tool. With it, we are very
easily able to demonstrate certain statistical results. The first of these is a matrix
explanation of distance, which we now develop.
Because x′ Ax has only squared terms x2i and product terms xi xk , it is called a
quadratic form. When a k × k symmetric matrix A is such that

0 ≤ x′ Ax

for all x′ = [x1 , x2 , . . . , xk ], both the matrix A and the quadratic form are said to be
nonnegative definite. If equality holds in (2-17) only for the vector x′ = [0, 0, . . . , 0],
then A or the quadratic form is said to be positive definite. In other words, A is
positive definite if
0 < x′ Ax
for all vectors x ̸= 0
Using the spectral decomposition, we can easily show that a k × k symmetric
matrix A is a positive definite matrix if and only if every eigenvalue of A is positive.
(See Exercise 2.17.) A is a nonnegative definite matrix if and only if all of its
eigenvalues are greater than or equal to zero.
Assume for the moment that the p elements x1 , x2 , . . . , xp of a vector x are
realizations of p random variables X1 , X2 , . . . , Xp . As we pointed out in Chapter 1,
we can regard these elements as the coordinates of a point in p-dimensional space,
and the "distance" of the point [x1 , x2 , . . . , xp ] to the origin can, and in this case
should, be interpreted in terms of standard deviation units. In this way, we can

Nguyễn Thị Nhung - Đại học Thăng Long

5.1. Revision 10

account for the inherent uncertainty (variability) in the observations. Points with
the same associated "uncertainty" are regarded as being at the same distance from
the origin. If we use the distance formula introduced in Chapter 1 [see Equation
(1-22)], the distance from the origin satisfies the general formula
If we use the distance formula introduced in Chapter 1 [see Equation (1 − 22) ],
the distance from the origin satisfies the general formula

( distance )2 = a11 x21 + a22 x22 + · · · +app x2p

+ 2 (a12 x1 x2 + a13 x1 x3 + · · · + ap−1,p xp−1 xp )

provided that (distance) 2 > 0 for all [x1 , x2 , . . . , xp ] ̸= [0, 0, . . . , 0]. Setting aij = aji ,
i ̸= j, i = 1, 2, . . . , p, j = 1, 2, . . . , p, we have
  
a11 a12 · · · a1p x1
 a21 a22 · · · a2p   x2 
  
2
0 < ( distance ) = [x1 , x2 , . . . , xp ]   .. .. . .   .. 
..   
 . . . .  . 
ap1 ap2 · · · app xp
or
0 < ( distance )2 = x′ Ax for x ̸= 0
From (2-19), we see that the p × p symmetric matrix A is positive definite. In sum,
distance is determined from a positive definite quadratic form x′ Ax. Conversely, a
positive definite quadratic form can be interpreted as a squared distance.
Comment. Let the square of the distance from the point x′ = [x1 , x2 , . . . , xp ]
to the origin be given by x′ Ax, where A is a p × p symmetric positive definite
matrix. Then the square of the distance from x to an arbitrary fixed point µ′ =
[µ1 , µ2 , . . . , µp ] is given by the general expression (x − µ)′ A(x − µ).
Expressing distance as the square root of a positive definite quadratic form allows
us to give a geometrical interpretation based on the eigenvalues and eigenvectors of
the matrix A. For example, suppose p = 2. Then the points x′ = [x1 , x2 ] of constant
distance c from the origin satisfy

x′ Ax = a11 x21 + a22 x22 + 2a12 x1 x2 = c2

By the spectral decomposition, as in Example 2.11,

2 2
A = λ1 e1 e′1 + λ2 e2 e′2 so x′ Ax = λ1 (x′ e1 ) + λ2 (x′ e2 )

Nguyễn Thị Nhung - Đại học Thăng Long

5.1. Revision 11

Now, c2 = λ1 y12 + λ2 y22 is an ellipse in y1 = x′ e1 and y2 = x′ e2 because λ1 , λ2 > 0

−1/2
when A is positive definite. (See Exercise 2.17.) We easily verify that x = cλ1 e1
 2
−1/2 −1/2
satisfies x′ Ax = λ1 cλ1 e′1 e1 = c2 . Similarly, x = cλ2 e2 gives the appropri-
ate distance in the e2 direction. Thus, the points at distance c lie on an ellipse whose
axes are given by the eigenvectors of A with lengths proportional to the reciprocals
of the square roots of the eigenvalues. The constant of proportionality is c. The
situation is illustrated in Figure 2.6.

√
If p > 2, the points x′ = [x1 , x2 , . . . , xp ] a constant distance c = x′ Ax from
the origin lie on hyperellipsoids c2 = λ1 (x′ e1 )2 + · · · + λp (x′ ep )2 , whose axes are
√
given by the eigenvectors of A. The half-length in the direction ei is equal to c/ λi ,
i = 1, 2, . . . , p, where λ1 , λ2 , . . . , λp are the eigenvalues of A.

A SQUARE-ROOT MATRIX

The spectral decomposition allows us to express the inverse of a square matrix in

terms of its eigenvalues and eigenvectors, and this leads to a useful square-root
matrix. Let A be a k × k positive definite matrix with the spectral decomposition

Nguyễn Thị Nhung - Đại học Thăng Long

5.1. Revision 12

A = ki=1 λi ei e′i . Let the normalized eigenvectors be the columns of another matrix
P

P = [e1 , e2 , . . . , ek ]. Then

k
X
A = λi ei e′i = PP
(k×k) (k×1)(1×k)(k×k)(k×k)(k×k) (P P
i=1

where PP′ = P′ P = I and Λ is the diagonal matrix

 
λ1 0 · · · 0
 0 λ2 · · · 0 
 
Λ =  .. .. . .  with λi > 0
.. 
(k×k)  . . . . 
0 0 · · · λk

The square-root matrix, of a positive matrix A,

k p
X
A 1/2
= λi ei e′i = PΛ1/2 P′
i=1

has the following properties:

′
1. A1/2 = A1/2 (that is, A1/2 is symmetric).

2. A1/2 A1/2 = A.

−1 Pk
3. A1/2 = i=1 √1λi ei e′i = PΛ−1/2 P′ , where Λ−1/2 is a diagonal matrix with
√
1/ λi as the i th diagonal element.

−1
4. A1/2 A−1/2 = A−1/2 A1/2 = I, and A−1/2 A−1/2 = A−1 , where A−1/2 = A1/2 .

5.1.2 Random vectors and matrices

A random vector is a vector whose elements are random variables. Similarly, a

random matrix is a matrix whose elements are random variables. The expected
value of a random matrix (or vector) is the matrix (vector) consisting of the expected
values of each of its elements. Specifically, let X = {Xij } be an n×p random matrix.

Nguyễn Thị Nhung - Đại học Thăng Long

5.1. Revision 13

Then the expected value of X, denoted by E(X), is the n × p matrix of numbers (if
they exist)  
E (X11 ) E (X12 ) · · · E (X1p )
 E (X21 ) E (X22 ) · · · E (X2p ) 
 
E(X) =   .. .. .. .. 
 . . . .


E (Xn1 ) E (Xn2 ) · · · E (Xnp )
where, for each element of the matrix, 2
R ∞


 x f (xij ) dxij
−∞ ij ij
if Xij is a continuous random variable with


 probability density function f (x )
ij ij
E (Xij ) = P
all xij xij pij (xij ) if Xij is a discrete random variable with





probability function pij (xij )


E(X + Y) = E(X) + E(Y)

E(AXB) = AE(X)B

5.1.3 Mean Vectors and Covariance Matrices

Suppose X′ = [X1 , X2 , . . . , Xp ] is a p × 1 random vector. Then each element of X

is a random variable with its own marginal probability distribution. (See Example
2.12.) T 69 marginal means µi and variances σi2 are defined as µi = E (Xi ) and
σi2 = E (Xi − µi )2 , i = 1, 2, . . . , p, respectively. Specifically,


 R∞ if Xi is a continuous random variable with probability
xi fi (xi ) dxi


−∞


 density function fi (xi )
µi = P
all xi xi pi (xi ) if Xi is a discrete random variable with probability





function pi (xi )



 ∞ (xi − µi )2 fi (xi ) dxi

 R if Xi is a continuous random variable
−∞
σi2 = with probability density function fi (xi )

2
 P
all xi (xi − µi ) pi (xi ) if Xi is a discrete random variable



It will be convenient in later sections to denote the marginal variances by σii rather
than the more traditional σi2 , and consequently, we shall adopt this notation.

Nguyễn Thị Nhung - Đại học Thăng Long

5.1. Revision 14

The behavior of any pair of random variables, such as Xi and Xk , is described

by their joint probability function, and a measure of the linear association between
them is provided by the covariance
σik = E (Xi − µi ) (Xk − µk )
R R
∞ ∞


 −∞ −∞
(xi − µi ) (xk − µk ) fik (xi , xk ) dxi dxk if Xi , Xk are continuous





 random variables with


 the joint density
= P
all xi all xk (xi − µi ) (xk − µk ) pik (xi , xk ) function fik (xi , xk )





if Xi , Xk are discrete






 random variable with
and µi and µk , i, k = 1, 2, . . . , p, are the marginal means. When i = k, the covariance
becomes the marginal variance.
More generally, the collective behavior of the p random variables X1 , X2 , . . . , Xp
or, equivalently, the random vector X′ = [X1 , X2 , . . . , Xp ], is described by a joint
probability density function f (x1 , x2 , . . . , xp ) = f (x). As we have already noted in
this book, f (x) will often be the multivariate normal density function. (See Chapter
4.) If the joint probability P [Xi ≤ xi and Xk ≤ xk ] can be written as the product
of the corresponding marginal probabilities, so that

P [Xi ≤ xi and Xk ≤ xk ] = P [Xi ≤ xi ] P [Xk ≤ xk ]

for all pairs of values xi , xk , then Xi and Xk are said to be statistically indepen-
dent. When Xi and Xk are continuous random variables with joint density fik (xi , xk )
and marginal densities fi (xi ) and fk (xk ), the independence condition becomes

fik (xi , xk ) = fi (xi ) fk (xk )

for all pairs (xi , xk ) The p continuous random variables X1 , X2 , . . . , Xp are mutually
statistically independent if their joint density can be factored as

f12···p (x1 , x2 , . . . , xp ) = f1 (x1 ) f2 (x2 ) · · · fp (xp )

for all p-tuples (x1 , x2 , . . . , xp ). Statistical independence has an important implica-

tion for covariance. The factorization in (2-28) implies that Cov (Xi , Xk ) = 0. Thus,

Cov (Xi , Xk ) = 0 if X1 and Xk are independent

Nguyễn Thị Nhung - Đại học Thăng Long

5.1. Revision 15

The converse of (2-29) is not true in general; there are situations where Cov (Xi , Xk ) =
0, but Xi and Xk are not independent. (See [2].)
The means and covariances of the p × 1 random vector X can be set out as
matrices. The expected value of each element is contained in the vector of means
µ = E(X), and the p variances σii and the p(p − 1)/2 distinct covariances σik (i < k)
are contained in the symmetric variance-covariance matrix Σ = E(X − µ)(X − µ)′ .
Specifically,    
E (X1 ) µ1
 E (X2 )   µ2 
   
E(X) =  ..  =  ..  = µ
  
 .   . 
E (Xp ) µp
 
σ11 σ12 · · · σ1p
σ21 σ22 · · · σ2p
 
 
Σ = Cov(X) =  .. .. .. . 

 . . . .. 

σp1 σp2 · · · σpp

We note that the computation of means, variances, and covariances for discrete
random variables involves summation (as in Examples 2.12 and 2.13), while analo-
gous computations for continuous random variables involve integration.
Because σik = E (Xi − µi ) (Xk − µk ) = σki , it is convenient to write the matrix
appearing in (2-31) as
 
σ11 σ12 · · · σ1p
 σ12 σ22 · · · σ2p 
 
′
Σ = E(X − µ)(X − µ) =  .  .. . . .. 
 . . . . . 

σ1p σ2p · · · σpp
We shall refer to µ and Σ as the population mean (vector) and population variance-
covariance (matrix), respectively.
The multivariate normal distribution is completely specified once the mean vector
µ and variance-covariance matrix Σ are given (see Chapter 4), so it is not surprising
that these quantities play an important role in many multivariate procedures.
It is frequently informative to separate the information contained in variances
σii from that contained in measures of association and, in particular, the measure

Nguyễn Thị Nhung - Đại học Thăng Long

5.1. Revision 16

of association known as the population correlation coefficient ρik . The correlation

coefficient ρik is defined in terms of the covariance σik and variances σii and σkk as
σik
ρik = √ √
σii σkk
The correlation coefficient measures the amount of linear association between the
random variables Xi and Xk . (See, for example, [2].)
Let the population correlation matrix be the p × p symmetric matrix
 
σ
√ σ11 √ √ σ12 √ · · · √ 1p √
 σ11σ σ11 σ11 σ22
σ
σ11 σpp
σ2p

 σ11 σ22 √σ22 √σ22 · · · √σ22 √σpp 
 √ 12√ 22 
ρ= .. .. .. .. 

 . . . . 

σ1p σ2p σpp
√ √
σ11 σpp
√ √
σ22 σpp
· · · √ √
σpp σpp
 
1 ρ12 · · · ρ1p
 ρ12 1 · · · ρ2p 
 
=  .. .. . . .. 
 . . . . 

ρ1p ρ2p · · · 1

and let the p × p standard deviation matrix be

 √ 
σ11 0 ··· 0
√
 0 σ22 · · · 0
 

V1/2 =  . . . ... .. 
 . .. .


√
0 0 ··· σpp

Then it is easily verified (see Exercise 2.23 ) that

V1/2 ρV1/2 = Σ

and

−1
−1
ρ = V1/2 Σ V1/2
That is, Σ can be obtained from V1/2 and ρ, whereas ρ can be obtained from Σ.
Moreover, the expression of these relationships in terms of matrix operations allows
the calculations to be conveniently implemented on a computer.
Partitioning the Covariance Matrix Often, the characteristics measured on in-
dividual trials will fall naturally into two or more groups. As examples, consider

Nguyễn Thị Nhung - Đại học Thăng Long

5.1. Revision 17

measurements of variables representing consumption and income or variables rep-

resenting personality traits and physical characteristics. One approach to handling
these situations is to let the characteristics defining the distinct groups be subsets
of the total collection of characteristics. If the total collection is represented by a
(p × 1)-dimensional random vector X, the subsets can be regarded as components
of X and can be sorted by partitioning X.
In general, we can partition the p characteristics contained in the p × 1 random
vector X into, for instance, two groups of size q and p − q, respectively. For example,
we can write
   
X1 
 µ1
 ..    .. 
 .  
  . 
  " #   " #
 X 
q 
 X(1)  µ 
 q  µ(1)
X=  qp − and µ = E(X) =  =

(2)
 Xq+1   X  µq+1  µ(2)
 ..   .. 
 
  
 .  
  . 

Xp µp


Partitioning the Covariance Matrix Often, the characteristics measured on in-

dividual trials will fall naturally into two or more groups. As examples, consider
measurements of variables representing consumption and income or variables rep-
resenting personality traits and physical characteristics. One approach to handling
these situations is to let the characteristics defining the distinct groups be subsets
of the total collection of characteristics. If the total collection is represented by a
(p × 1)-dimensional random vector X, the subsets can be regarded as components
of X and can be sorted by partitioning X.
In general, we can partition the p characteristics contained in the p × 1 random
vector X into, for instance, two groups of size q and p − q, respectively. For example,

Nguyễn Thị Nhung - Đại học Thăng Long

5.1. Revision 18

we can write
 
X1 

 .. 


 . 

   
 
 µ1 µq
 Xq
 
 ..
  " (1) #
µq+1
   
 p − qu =  . = µ
    
X=  ······
 and µ = E(X) =  ..
  (1)
 X
 
.
 µ(2)
 Xq+1
 
   

 ..


 X(2) µp

 . 




Xp 

Making use of the partitioning in Equation (2-38), we can easily demonstrate

that (X − µ)(X − µ)′
 ! !′ ! 
 ′
(2)
 X(1) −µ(1) X(1) −µ(1) X(1) −µ(1) X(1×(p−q)) − µ(2) 
 (q×1) (1×q) (q×1) 
= ! ′   
′
  
(µ )
(2)
  
(2) (2) (1) (1) (2) (2)
X((p−q)×1) − µ X −µ X((p−q)×1) X(1×(p−q)) − µ
 
(1×q)

and consequently,
" #
′ q Σq11 Σp−q
12
Σ(p×1 = E(X − µ)(X − µ) = p−q
(p×p) Σ21 Σ22
 
σ11 ··· σ1q σ1,q+1 ··· σ1p
 .. .. .. .. .. .. 
 .
 . . . . . 

 σ
q1 · · · σqq σq,q+1 · · · σqp 
=
 

 σq+1,1 · · · σq+1,q σq+1,q+1 · · · σq+1,p 
 .. .. .. .. .. ..
 
 . . . . . .


σp1 · · · σpq σp,q+1 · · · σpp

Note that Σ12 = Σ′21 . The covariance matrix of X(1) is Σ11 , that of X(2) is Σ22 ,
and that of elements from X(1) and X(2) is Σ12 (or Σ21 ).

Nguyễn Thị Nhung - Đại học Thăng Long

5.1. Revision 19

5.1.4 The Mean Vector and Covariance Matrix for Linear

Combinations of Random Variables

Recall that if a single random variable, such as X1 , is multiplied by a constant c,

then
E (cX1 ) = cE (X1 ) = cµ1
and
Var (cX1 ) = E (cX1 − cµ1 )2 = c2 Var (X1 ) = c2 σ11
If X2 is a second random variable and a and b are constants, then, using additional
properties of expectation, we get

Cov (aX1 , bX2 ) = E (aX1 − aµ1 ) (bX2 − bµ2 )

= abE (X1 − µ1 ) (X2 − µ2 )
= ab Cov (X1 , X2 ) = abσ12

Finally, for the linear combination aX1 + bX2 , we have

E (aX1 + bX2 ) = aE (X1 ) + bE (X2 ) = aµ1 + bµ2

" #2
Var (aX1 + bX2 ) = E (|{z} +bX2 ) − (aµ1 + bµ2 )
1

= E [a (X1 − µ1 ) + b (X2 − µ2 )]2

= E a2 (X1 − µ1 )2 + b2 (X2 − µ2 )2 + 2ab (X1 − µ1 ) (X2 − µ2 )
 

= a2 Var (X1 ) + b2 Var (X2 ) + 2ab Cov (X1 , X2 )

= a2 σ11 + b2 σ22 + 2abσ12

Similarly, E (aX1 + bX2 ) = aµ1 + bµ2 can be expressed as

" #
h i µ
1
a b = c′ µ
µ2

If we let " #
σ11 σ12
Σ=
σ12 σ22
be the variance-covariance matrix of X, Equation (2-41) becomes

Var (aX1 + bX2 ) = Var (c′ X) = c′ Σc

Nguyễn Thị Nhung - Đại học Thăng Long

5.1. Revision 20

since " #" #

h i σ11 σ12 a
c′ Σc = a b = a2 σ11 + 2abσ12 + b2 σ22
σ12 σ22 b

The linear combination c′ X = c1 X1 + · · · + cp Xp has

mean = E (c′ X) = c′ µ
variance = Var (c′ X) = c′ Σc

where µ = E(X) and Σ = Cov(X)

In general, consider the q linear combinations of the p random variables X1 , . . . , Xp
:
Z1 = c11 X1 + c12 X2 + · · · + c1p Xp
Z2 = c21 X1 + c22 X2 + · · · + c2p Xp
..
.
..
.
Zq = cq1 X1 + cq2 X2 + · · · + cqp Xp
or      
Z1 c11 c12 · · · c1p X1
Z2 c21 c22 · · · c2p  X2
     
    
Z= .. = .. .. .. .   ..  = CX

 .
 
  . . . ..  (q×p) 
 .


Zq×1) cq1 cq2 · · · cqp Xp

The linear combinations Z = CX have

µZ = E(Z) = E(CX) = CµX

ΣZ = Cov(Z) = Cov(CX) = CIX C′

where µX and ΣX are the mean vector and variance-covariance matrix of X, respec-
tively. (See Exercise 2.28 for the computation of the off-diagonal terms in CΣX C′ .)
Partitioning the Sample Mean Vector and Covariance Matrix Many of the ma-
trix results in this section have been expressed in terms of population means and
variances (covariances). The results in (2-36), (2-37), (2-38), and (2-40) also hold if
the population quantities are replaced by their appropriately defined sample coun-
terparts.

Nguyễn Thị Nhung - Đại học Thăng Long

5.1. Revision 21

Let x′ = [x̄1 , x̄2 , . . . , x̄p ] be the vector of sample averages constructed from n
observations on p variables X1 , X2 , . . . , Xp , and let
 
s11 · · · s1p
 . . 
Sn =  .. . . . .. 
s1p · · · spp
Pn
(xj1 − x̄1 )2
 1 1
Pn 
n j=1 ··· n
(xj1 − x̄1 ) (xjp − x̄p )
j=1
.. .. ..
= .
 
. . 
1
Pn 1
Pn 2
n j=1 (xj1 − x̄1 ) (xjp − x̄p ) · · · n j=1 (xjp − x̄p )

be the corresponding sample variance-covariance matrix.

The sample mean vector and the covariance matrix can be partitioned in order
to distinguish quantities corresponding to groups of variables. Thus,
 
x̄1
 .. 

 . 
 " #
 x̄q  x(1)
x = =
 
(p×1)  . . . x̄q+1  . . . x(2)
..
 
.
 
 
x̄p

and  
s11 ··· s1q s1,q+1 ··· s1p
 .. ... .. .. ... .. 

 . . . . 

 sq1 · · · sqq sq,q+1 · · · sqp 
Sn = 
 

(p×p)  . . . sq+1,1 · · · sq+1,q sq+1,q+1 · · · sq+1,p 
.. ... .. .. ... ..
 
. . . .
 
 
sp1 · · · spq sp,q+1 · · · spp
" #
Sq11 S12
= qp−q
. . . S21 S22
where x(1) and x(2) are the sample mean vectors constructed from observations
x(1) = [x1 , . . . , xq ]′ and x(2) = [xq+1 , . . . , xp ]′ , respectively; S11 is the sample covari-
ance matrix computed from observations x(1) ; S22 is the sample covariance matrix
computed from observations x(2) ; and S12 = S′21 is the sample covariance matrix for
elements of x(1) and elements of x(2) .

Nguyễn Thị Nhung - Đại học Thăng Long

5.1. Revision 22

Theorem 1 (The Spectral Decomposition). Let A be a k × k symmetric matrix.

Then A can be expressed in terms of its k eigenvalue-eigenvector pairs (λi , ei ) as
k
X
A= λi ei e′i
i=1

The ideas that lead to the spectral decomposition can be extended to provide a
decomposition for a rectangular, rather than a square, matrix. If A is a rectangular
matrix, then the vectors in the expansion of A are the eigenvectors of the square
matrices AA′ and A′ A.

Theorem 2 (Singular-Value Decomposition). Let A be an m × k matrix of real

numbers. Then there exist an m × m orthogonal matrix U and a k × k orthogonal
matrix V such that
A = UΛV′
where the m × k matrix Λ has (i, i) entry λi ≥ 0 for i = 1, 2, . . . , min(m, k) and the
other entries are zero. The positive constants λi are called the singular values of A.

The singular-value decomposition can also be expressed as a matrix expansion

that depends on the rank r of A. Specifically, there exist r positive constants
λ1 , λ2 , . . . , λr , r orthogonal m × 1 unit vectors u1 , u2 , . . . , ur , and r orthogonal k × 1
unit vectors v1 , v2 , . . . , vr , such that
r
X
A= λi ui vi′ = Ur Λr Vr′
i=1

where Ur = [u1 , u2 , . . . , ur ] , Vr = [v1 , v2 , . . . , vr ], and Λr is an r×r diagonal matrix

with diagonal entries λi . Here AA′ has eigenvalue-eigenvector pairs (λ2i , ui ), so

AA′ ui = λ2i ui

with λ21 , λ22 , . . . , λ2r > 0 = λ2r+1 , λ2r+2 , . . . , λ2m (for m > k ). Then vi = λ−1 ′
i A ui .
Alternatively, the vi are the eigenvectors of A′ A with the same nonzero eigenvalues
λ2i .
The matrix expansion for the singular-value decomposition written in terms of
the full dimensional matrices U, V, Λ is

A = U
(m×k) (m×m)(m×k)(k×k)(V′

Nguyễn Thị Nhung - Đại học Thăng Long

5.1. Revision 23

where U has m orthogonal eigenvectors of AA′ as its columns, V has k orthogonal

eigenvectors of A′ A as its columns, and Λ is specified in Result 2A.15. For example,
let " #
3 1 1
A=
−1 3 1
Then  
" # 3 −1 " #
3 1 1 11 1
AA′ =  1 3 =
 
−1 3 1 1 11
1 1
You may verify that the eigenvalues γ = λ2 of AA′ satisfy the equation γ 2 − 22γ +
120 = (γ − 12)(γ − 10), and consequently, the eigenvalues hare γ1 = λi21 = 12 and
γ2 = λ22 = 10. The corresponding eigenvectors are u′1 = √12 √12 and u′2 =
h i
√1 −1
2
√
2
, respectively.
Also,    
3 −1 " # 10 0 2
3 1 1
A′ A =  1 3  =  0 10 4 
   
−1 3 1
1 1 2 4 2
so |A′ A − γI| = −γ 3 − 22γ 2 − 120γ = −γ(γ − 12)(γ − 10), and the eigenvalues are
γ1 = λ21 = 12, γ2 = λ22 = 10, and γ3 = λ23 = 0. The nonzero eigenvalues are the same
as those of AA′ . A computer calculation gives the eigenvectors
h i h i h i
v1′ = √16 √26 √16 , v2′ = √25 √ −1
5
0 , and v ′
3 = √1
30
√2
30
−5
√
30
.

Eigenvectors v1 and v2 can be verified by checking:

     
10 0 2 1 1
′  1   1  
A Av1 =  0 10 4  √  2  = 12 √  2  = λ21 v1

6 6
2 4 2 1 1
     
10 0 2 2 2
′  1  1 
A Av2 =  0 10 4  √  −1  = 10 √  −1  = λ22 v2
  
5 5
2 4 2 0 0
√ √
Taking λ1 = 12 and λ2 = 10, we find that the singular-value decomposition of

Nguyễn Thị Nhung - Đại học Thăng Long

5.1. Revision 24

A is " #
3 1 1
A=
−1 3 1
" # " #
√ √1 h i √ √1 h
−1
i
= 12 2 √1 √2 √1 + 10 2 √2 √ 0
√1 6 6 6 −1
√ 5 5
2 2
The equality may be checked by carrying out the operations on the right-hand
side. The singular-value decomposition is closely connected to a result concerning
the approximation of a rectangular matrix by a lower-dimensional matrix, due to
Eckart and Young ([3]). If a m × k matrix A is approximated by B, having the same
dimension but lower rank, the sum of squared differences
m X
X k
(aij − bij )2 = tr [(A − B)(A − B)′ ]
i=1 j=1

Theorem 3. Let A be an m × k matrix of real numbers with m ≥ k and singular

value decomposition UΛV′ . Let s < k = rank(A). Then
s
X
B= λi ui vi′
i=1

is the rank-s least squares approximation to A. It minimizes

tr [(A − B)(A − B)′ ]

over all m × k matrices B having rank no greater than s. The minimum value, or
error of approximation, is ki=s+1 λ2i .
P

To establish this result, we use UU′ = Im and VV′ = Ik to write the sum of
squares as

tr [(A − B)(A − B)′ ] = tr [UU′ (A − B)V′ (A − B)′ ]

= tr [U′ (A − B)V′ (A − B)′ U]
m X
X k m
X 2
XX
′ 2 2
= tr [(Λ − C)(Λ − C) ] = (λij − cij ) = (λi − cii ) +
i=1 j=1 i=1 i̸=j ij

where C = U′ BV. Clearly, the minimum occurs when cij = 0 for i ̸= j and cii = λi
for the s largest singular values. The other cii = 0. That is, UBV′ = Λs or B =
Ps ′
i=1 λi ui vi .

Nguyễn Thị Nhung - Đại học Thăng Long

5.2. Linear Algebra - linalg in Python 25

Theorem 4. Let A be an m × k matrix of real numbers with m ≥ k and singular

value decomposition UΛV′ . Let s < k = rank(A). Then
s
X
B= λi ui vi′
i=1

is the rank-s least squares approximation to A. It minimizes

tr [(A − B)(A − B)′ ]

over all m × k matrices B having rank no greater than s. The minimum value, or
error of approximation, is ki=s+1 λ2i .
P

To establish this result, we use UU′ = Im and VV′ = Ik to write the sum of
squares as

tr [(A − B)(A − B)′ ] = tr [UU′ (A − B)V′ (A − B)′ ]

= tr [U′ (A − B)V′ (A − B)′ U]
m X
X k m
X 2
XX
′ 2 2
= tr [(Λ − C)(Λ − C) ] = (λij − cij ) = (λi − cii ) +
i=1 j=1 i=1 i̸=j ij

where C = U′ BV. Clearly, the minimum occurs when cij = 0 for i ̸= j and cii = λi
for the s largest singular values. The other cii = 0. That is, UBV′ = Λs or B =
Ps ′
i=1 λi ui vi .

5.2 Linear Algebra - linalg in Python

5.2.1 Matrix Operators

import numpy as np
import [Link] as la

A = [Link]([[1,2,-1], [2,0,1], [0,1,1]])

A
matrix([[ 1, 2, -1],
[ 2, 0, 1],

Nguyễn Thị Nhung - Đại học Thăng Long

5.2. Linear Algebra - linalg in Python 26

[ 0, 1, 1]])
B = [Link]([[2,0,-1], [-1,0,-1], [1,1,1]])
B
matrix([[ 2, 0, -1],
[-1, 0, -1],
[ 1, 1, 1]])

#Addition
A + B
matrix([[ 3, 2, -2],
[ 1, 0, 0],
[ 1, 2, 2]])

#Subtraction
A - B
matrix([[-1, 2, 0],
[ 3, 0, 2],
[-1, 0, 0]])

#Multiplication by a scalar
5*A
matrix([[ 5, 10, -5],
[10, 0, 5],
[ 0, 5, 5]])

#Multiplication of two matrices

A*B
matrix([[-1, -1, -4],
[ 5, 1, -1],
[ 0, 1, 0]])

#Determinant of a matrix
[Link](A)
-6.999999999999999

Nguyễn Thị Nhung - Đại học Thăng Long

5.2. Linear Algebra - linalg in Python 27

# The inverse matrix

[Link](A)
matrix([[ 0.14285714, 0.42857143, -0.28571429],
[ 0.28571429, -0.14285714, 0.42857143],
[-0.28571429, 0.14285714, 0.57142857]])

#Rank of a matrix
la.matrix_rank(A)
3

#Power of a matrix
la.matrix_power(A, 5)
matrix([[ 47, 53, -18],
[ 70, 29, 1],
[ 34, 35, -4]])

#Eigen values of a matrix

M = [Link]([[9, 1], [1, 9]])
M
matrix([[9, 1],
[1, 9]])

[Link](M)
(array([10., 8.]),
matrix([[ 0.70710678, -0.70710678],
[ 0.70710678, 0.70710678]]))

#Trace of a matrix
[Link](M)
18

Nguyễn Thị Nhung - Đại học Thăng Long

5.2. Linear Algebra - linalg in Python 28

Find the solution of this equations


9x + x = 2
1 2
x1 + 9x2 = 5

b = [Link]([2, 5])

sol = [Link](M, bsol = [Link](M, b)

sol
array([0.1625, 0.5375])

[Link]

Signature: [Link](a, full_matrices=True, compute_uv=True, hermitian=False)

Docstring:
Singular Value Decomposition.

When ‘a‘ is a 2D array, it is factorized as ‘‘u @ [Link](s) @ vh

= (u * s) @ vh‘‘, where ‘u‘ and ‘vh‘ are 2D unitary arrays and ‘s‘ is a 1D
array of ‘a‘’s singular values. When ‘a‘ is higher-dimensional, SVD is
applied in stacked mode as explained below.

Parameters
----------
a : (..., M, N) array_like
A real or complex array with ‘‘[Link] >= 2‘‘.
full_matrices : bool, optional
If True (default), ‘u‘ and ‘vh‘ have the shapes ‘‘(..., M, M)‘‘ and
‘‘(..., N, N)‘‘, respectively. Otherwise, the shapes are
‘‘(..., M, K)‘‘ and ‘‘(..., K, N)‘‘, respectively, where
‘‘K = min(M, N)‘‘.
compute_uv : bool, optional
Whether or not to compute ‘u‘ and ‘vh‘ in addition to ‘s‘. True
by default.
hermitian : bool, optional

Nguyễn Thị Nhung - Đại học Thăng Long

5.3. Exercises 29

If True, ‘a‘ is assumed to be Hermitian (symmetric if real-valued),

enabling a more efficient method for finding singular values.
Defaults to False.

Returns
-------
u : { (..., M, M), (..., M, K) } array
Unitary array(s). The first ‘‘[Link] - 2‘‘ dimensions have the same
size as those of the input ‘a‘. The size of the last two dimensions
depends on the value of ‘full_matrices‘. Only returned when
‘compute_uv‘ is True.
s : (..., K) array
Vector(s) with the singular values, within each vector sorted in
descending order. The first ‘‘[Link] - 2‘‘ dimensions have the same
size as those of the input ‘a‘.
vh : { (..., N, N), (..., K, N) } array
Unitary array(s). The first ‘‘[Link] - 2‘‘ dimensions have the same
size as those of the input ‘a‘. The size of the last two dimensions
depends on the value of ‘full_matrices‘. Only returned when
‘compute_uv‘ is True.

5.3 Exercises

Problem 1. Give the definition of dependent and independent vectors. Take an

example.

Problem 2. State the definition of the projection (or shadow) of a vector x on a

vector y and the length of the projection.

Problem 3. Give the definition of orthogonal matrices. Take an example.

Problem 4. Give the definition for eigenvalues and eigenvector of a square matrix
A.

Problem 5. What is a positive definite matrix? How can we check weather a matrix
is positive definite?

Nguyễn Thị Nhung - Đại học Thăng Long

5.3. Exercises 30

Problem 6. State the definition of square-root matrix of a positive definite matrix.

Give an example.

Problem 7. Write the formular for the means, covariances, correlation matrix of a
random vector X.

Problem 8. State the theorem of The spectral decomposition and Singular value
decomposition.

Problem 9. State the mean vector and covariance matrix for linear combinations
of random variables.
h i h i
′ ′
Problem 10. Let x = 5, 1, 3 and y = −1, 3, 1 .

a. Graph the two vectors.

b. Find (i) the length of x, (ii) the angle between x and y, and (iii) the projection
of y on x.

c. Since x̄ = 3 and ȳ = 1, graph [5 − 3, 1 − 3, 3 − 3] = [2, −2, 0] and [−1 − 1, 3 −

1, 1 − 1] = [−2, 2, 0]

Problem 11. Given the matrices

   
" # 4 −3 5
−1 3
A= , B =  1 −2  , and C =  −4 
   
4 2
−2 0 2

perform the indicated multiplications.

a. 5A

b. BA

c. A′ B′

d. C′ B

e. Is AB defined?

Nguyễn Thị Nhung - Đại học Thăng Long

5.3. Exercises 31

Problem 12. Verify the following properties of the transpose when

" # " # " #
2 1 1 4 2 1 4
A= , B= , and C =
1 3 5 0 3 3 2

a. (A′ )′ = A
′
b. (C′ )−1 = (C−1 )

c. (AB)′ = B′ A′

d. For general A and B , (AB)′ = B′ A′ .

(m×k) (k×ℓ)

Problem 13. When A−1 and B−1 exist, prove each of the following.

′
a. (A′ )−1 = (A−1 )

b. (AB)−1 = B−1 A−1

Problem 14. Check that " #

5 12
13 13
Q=
− 12
13
5
13

is an orthogonal matrix.

Problem 15. Let " #

9 −2
A=
−2 6

a. Is A symmetric?

b. Show that A is positive definite.

c. Determine the eigenvalues and eigenvectors of A.

d. Write the spectral decomposition of A.

e. Find A−1 .

f. Find the eigenvalues and eigenvectors of A−1 .

Nguyễn Thị Nhung - Đại học Thăng Long

5.3. Exercises 32

Problem 16. Given the matrix

" #
1 2
A= .
2 −2

a. Find the eigenvalues λ1 and λ2 and the associated normalized eigenvectors e1

and e2 . Determine the spectral decomposition (2-16) of A.

b. Find A−1 .

c. Compute the eigenvalues and eigenvectors of A−1 .

d. Write the spectral decomposition of A−1 , and compare it with that of A from
a.

Problem 17. Consider the matrices

" # " #
4 4.001 4 4.001
A= and B =
4.001 4.002 4.001 4.002001

These matrices are identical except for a small difference in the (2, 2) position.
Moreover, the columns of A (and B ) are nearly linearly dependent. Show that
.
A−1 = (−3)B−1 . Consequently, small changes-perhaps caused by rounding, can
give substantially different inverses.

Problem 18. Show that the determinant of the p × p diagonal matrix A = {aij }
with aij = 0, i ̸= j, is given by the product of the diagonal elements; thus, |A| =
a11 a22 · · · app .

Problem 19. Show that the determinant of a square symmetric p × p matrix A can
be expressed as the product of its eigenvalues λ1 , λ2 , . . . , λp ; that is, |A| = pi=1 λi .
Q

Problem 20. Show that |Q| = +1 or −1 if Q is a p × p orthogonal matrix.

Problem 21. Show that Q′ A Q and A have the same eigenvalues if

(p×p)(p×p) (p×p)(p×p) (p×p)
Q is orthogonal.

Problem 22. A quadratic form x′ Ax is said to be positive definite if the matrix A

is positive definite. Is the quadratic form 3x21 + 3x22 − 2x1 x2 positive definite?

Nguyễn Thị Nhung - Đại học Thăng Long

5.3. Exercises 33

Problem 23. Consider an arbitrary n×p matrix A. Then A′ A is a symmetric p×p

matrix. Show that A′ A is necessarily nonnegative definite.

Problem 24. Prove that every eigenvalue of a k × k positive definite matrix A is

positive.

Problem 25. Consider the sets of points (x1 , x2 ) whose "distances" from the origin
are given by
√
c2 = 4x21 + 3x22 − 2 2x1 x2
for c2 = 1 and for c2 = 4. Determine the major and minor axes of the ellipses
of constant distances and their associated lengths. Sketch the ellipses of constant
distances and comment on their positions. What will happen as c2 increases?
P √
Problem 26. Let A1/2 = m i=1 λi ei e′i = PΛ1/2 P′ , where PPP′ = P′ P = I. (The
(m×m)
λi ’s and the ei ’s are the eigenvalues and associated normalized eigenvectors of the
matrix A.) Show the following properties (1)-(4) of the square-root matrix.

′
1. A1/2 = A1/2 (that is, A1/2 is symmetric).

2. A1/2 A1/2 = A.

−1 Pk
3. A1/2 = i=1 √1λi ei e′i = PΛ−1/2 P′ , where Λ−1/2 is a diagonal matrix with
√
1/ λi as the i th diagonal element.

−1
4. A1/2 A−1/2 = A−1/2 A1/2 = I, and A−1/2 A−1/2 = A−1 , where A−1/2 = A1/2 .

Problem 27. Determine the square-root matrix A1/2 , using the matrix A in Exer-
cise 11. Also, determine A−1/2 , and show that A1/2 A−1/2 = A−1/2 A1/2 = I.

Problem 28 (See Singular-value decomposition). Using the matrix

 
1 1
A =  2 −2 
 

2 2

a. Calculate A′ A and obtain its eigenvalues and eigenvectors.

b. Calculate AA′ and obtain its eigenvalues and eigenvectors. Check that the
nonzero eigenvalues are the same as those in part (a).

Nguyễn Thị Nhung - Đại học Thăng Long

5.3. Exercises 34

c. Obtain the singular-value decomposition of A.

Problem 29 (See Singular-value decomposition). Using the matrix

" #
4 8 8
A=
3 6 −9

a. Calculate AA′ and obtain its eigenvalues and eigenvectors.

b. Calculate A′ A and obtain its eigenvalues and eigenvectors. Check that the
nonzero eigenvalues are the same as those in part a.

c. Obtain the singular-value decomposition of A.

−1
−1
Problem 30. Verify the relationships V1/2 ρV1/2 = Σ and ρ = V1/2 Σ V1/2 ,
where Σ is the p × p population covariance matrix [Equation (2-32)], ρ is the p × p
population correlation matrix [Equation (2-34)], and V1/2 is the population standard
deviation matrix [Equation (2-35)].

Problem 31. Let X have covariance matrix

 
4 0 0
Σ= 0 9 0 
 

0 0 1

Find

a. Σ−1

b. The eigenvalues and eigenvectors of Σ.

c. The eigenvalues and eigenvectors of Σ−1 .

Problem 32. Let X have covariance matrix

 
25 −2 4
Σ =  −2 4 1 
 

4 1 9

a. Determine ρ and V1/2 .

Nguyễn Thị Nhung - Đại học Thăng Long

5.3. Exercises 35

b. Multiply your matrices to check the relation V1/2 ρV1/2 = Σ.

c. Find ρ13 .

d. Find the correlation between X1 and 21 X2 + 12 X3 .

Problem 33. Derive expressions for the mean and variances of the following linear
combinations in terms of the means and covariances of the random variables X1 , X2 ,
and X3 .

a. X1 − 2X2

b. −X1 + 3X2

c. X1 + X2 + X3

d. X1 + 2X2 − X3

e. 3X1 − 4X2 if X1 and X2 are independent random variables.

Problem 34. Show that Cov (c11 X1 + c12 X2 + · · · + c1p Xp , c21 X1 + c22 X2 + · · · + c2p Xp ) =
c′1 Σx c2 where c′1 = [c11 , c12 , . . . , c1p ] and c′2 = [c21 , c22 , . . . , c2p ]. This verifies the off-
diagonal elements CΣX C′ in (2-45) or diagonal elements if c1 = c2

Problem 35. Consider the arbitrary random vector X′ = [X1 , X2 , X3 , X4 , X5 ] with

mean vector µ′ = [µ1 , µ2 , µ3 , µ4 , µ5 ]. Partition X into
" #
(1)
X
X=
X(2)

where  
" # X3
X1
X(1) = and X(2) =  X4 
 
X2
X5
Let Σ be the covariance matrix of X with general element σik . Partition Σ into the
covariance matrices of X(1) and X(2) and the covariance matrix of an element of X(1)
and an element of X(2)

Nguyễn Thị Nhung - Đại học Thăng Long

5.3. Exercises 36

Problem 36. You are given the random vector X′ = [X1 , X2 , X3 , X4 ] with mean
vector µ′X = [4, 3, 2, 1] and variance-covariance matrix
 
3 0 2 2
 0 1 1 0 
ΣX = 
 

 2 1 9 −2 
2 0 −2 4

Partition X as  
X1
  (1) 
X2 X

 
  
X=
 ...  =  ... 
 
(2)
X3 X
 
 
X4
Let " #
h i 1 −2
A= 1 2 and B =
2 −1
and consider the linear combinations AX(1) and BX(2) . Find


a. E X(1)


b. E AX(1)


c. Cov X(1)
 
d. Cov AX(1)


e. E X(2)
 
f. E BX(2)


g. Cov X(2)
 
h. Cov BX(2)


i. Cov X(1) , X(2)
 
h. Cov AX(1) , BX(2)

Nguyễn Thị Nhung - Đại học Thăng Long