Multivariate Data Analysis

Introduction

Prof. Gabriel Asare Okyere

May 17, 2023

Prof. Gabriel Asare Okyere (PhD) Introduction to Multivariate Data Analysis May 17, 2023 1 / 41
 Introduction

Outline of Presentation

1 Introduction

2 Organization of Data and Notation

3 Descriptive Statistics

4 Distance

Prof. Gabriel Asare Okyere (PhD) Introduction to Multivariate Data Analysis May 17, 2023 2 / 41
 Introduction

Introduction to Multivariate Data Analysis

In many observational studies, observations are collected

simultaneously on more than one variable on each experimental
unit.
Multivariate analysis is the collection of methods that can be
used to analyze these multiple measurements.
The main idea is to exploit potential correlations among multiple
measurements to improve inference.
Some multivariate techniques rely on an underlying probability
model, typically the multivariate normal distribution. Others are
’model-free’

Prof. Gabriel Asare Okyere (PhD) Introduction to Multivariate Data Analysis May 17, 2023 3 / 41
 Introduction

Objectives of Multivariate Analysis

Dimensionality reduction: Can we reduce the dimensionality of

the problem by considering a small number of (linear)
combinations of a large number of measurements without losing
important information?

Grouping : Identify groups of ’similar’ units using a common set

of measured traits.

Classification: Classify units into previously defined groups using

a common set of measured traits.

Prof. Gabriel Asare Okyere (PhD) Introduction to Multivariate Data Analysis May 17, 2023 4 / 41
 Introduction

Objectives of Multivariate Analysis

Dependence among variables: What is the nature of associations

among variables?

Prediction: If variables are associated, then we might be able to

predict the value of some of them given information on the
others.

Hypothesis testing : Are differences in sets of response means for

two or more groups large enough to be distinguished from
sampling variation?

Prof. Gabriel Asare Okyere (PhD) Introduction to Multivariate Data Analysis May 17, 2023 5 / 41
 Introduction

Examples: Classification and Grouping

An insurance company wishes to group customers with respect
to products purchased and demographic variables to target
marketing efforts at different groups.

A marketing company examines traits of people who respond or

fail to respond to a mass mailing.

The IRS of Ghana uses data collected from tax returns (income,
amount withheld, deductions, contributions to charity, age) to
classify taxpayers into two groups: those who will be audited and
those who will not.
Prof. Gabriel Asare Okyere (PhD) Introduction to Multivariate Data Analysis May 17, 2023 6 / 41
 Introduction

Examples: Hypothesis Testing

A transportation company wants to know if means for gasoline

mileage, repair costs, downtime due to repairs differ for different
truck models.

The Environmental Protection Agency (EPA) of Ghana wants to

know if different tillage practices lead to different patterns of
nitrate concentrations in nearby waterways?

Prof. Gabriel Asare Okyere (PhD) Introduction to Multivariate Data Analysis May 17, 2023 7 / 41
 Introduction

Examples: Dimensionality Reduction

An index of consumer satisfaction with new car ownership can

be constructed from dozens of questions on a survey.

Wildlife ecologists can construct a few indices of habitat

preference from measurements of dozens of features of nesting
sites selected by a certain bird species.

Prof. Gabriel Asare Okyere (PhD) Introduction to Multivariate Data Analysis May 17, 2023 8 / 41
 Organization of Data and Notation

Outline of Presentation

1 Introduction

2 Organization of Data and Notation

3 Descriptive Statistics

4 Distance

Prof. Gabriel Asare Okyere (PhD) Introduction to Multivariate Data Analysis May 17, 2023 9 / 41
 Organization of Data and Notation

Organization of Data and Notation

We will use n to denote the number of individuals or units in our
sample and use p to denote the number of variables measured
on each unit.

If p = 1, then we have the usual univariate setting.

xik is the value of the k − th measurement on the i − th unit or

observation.
For the i − th unit we have measurements

xi1 , xi2 , . . . , xip

Prof. Gabriel Asare Okyere (PhD) Introduction to Multivariate Data Analysis May 17, 2023 10 / 41
 Organization of Data and Notation

Organization of Data and Notation

We often collect all measurements taken on the i − th unit into

a column vector. If five measurements are taken on the i − th
unit, we would have  
xi1
xi2 
 
xi3 
xi =  
xi4 
xi5

Prof. Gabriel Asare Okyere (PhD) Introduction to Multivariate Data Analysis May 17, 2023 11 / 41
 Organization of Data and Notation

Organization of Data and Notation

We often display measurements from a sample of n units in

matrix form:  
x11 x12 . . . x1p
x21 x22 . . . x2p 
Xn×p =  ..
 
.. . . .. 
 . . . . 
xn1 xn2 . . . xnp
where X is a matrix with n rows (one for each unit) and p
columns (one for each measured trait or variable).

Prof. Gabriel Asare Okyere (PhD) Introduction to Multivariate Data Analysis May 17, 2023 12 / 41
 Descriptive Statistics

Outline of Presentation

1 Introduction

2 Organization of Data and Notation

3 Descriptive Statistics

4 Distance

Prof. Gabriel Asare Okyere (PhD) Introduction to Multivariate Data Analysis May 17, 2023 13 / 41
 Descriptive Statistics

Descriptive Statistics
The sample mean of the kth variable (k = 1, . . . , p) is computed
as: n
1X
x̄k = xik
n i=1
The sample variance of the kth variable usually computed as:
n
1 X
sk2 = (xik − x̄k )2
n − 1 i=1

and the sample standard deviation is given by

q
sk = sk2

Prof. Gabriel Asare Okyere (PhD) Introduction to Multivariate Data Analysis May 17, 2023 14 / 41
 Descriptive Statistics

Example

Two measurements x1 ; x2 made at the same position on each of

3 cans of food, resulted in the following X-matrix:
 
4 1
X = −1 3
3 5

Find the sample mean vector x̄ and covariance matrix S.

Prof. Gabriel Asare Okyere (PhD) Introduction to Multivariate Data Analysis May 17, 2023 15 / 41
 Descriptive Statistics

Solution


4 1
X = −1 3 = [x1 , x2 , x3 ]T
3 5
3        
1X 1 4 −1 3 2
x̄ = xi = + + =
n i=1 3 1 3 5 3
 
2 −2
X ′ = −3 0 
1 2
n
1 X 1 
s12 (xi1 − x̄1 )2 = (4 − 2)2 + (−1 − 2)2 + (3 − 2)2

=
n − 1 i=1 3−1
√
s12 = 7 and s1 = 7
Prof. Gabriel Asare Okyere (PhD) Introduction to Multivariate Data Analysis May 17, 2023 16 / 41
 Descriptive Statistics

Descriptive Statistics
NB: Sometimes, the variance is defined with a denominator of n
instead of n − 1, and this will be clear from the notation.
We often use skk to denote the sample variance for the kth
variable. Thus,
sk2 = skk
The sample covariance between variable k and variable j is
computed as
n
1 X
sjk = (xij − x̄j )(xik − x̄k )
n − 1 i=1

If variables k and j are independent, the population covariance

will be exactly zero, but the sample covariance will vary about
zero.
Prof. Gabriel Asare Okyere (PhD) Introduction to Multivariate Data Analysis May 17, 2023 17 / 41
 Descriptive Statistics

Descriptive Statistics

The sample correlation between variables k and j is defined as

sjk
rjk = √ √
sjj skk

rjk is between −1 and 1.

rjk = rkj
The sample correlation is the same whether n or n − 1 is used as
the divisor in evaluating sample variances and covariances.

Prof. Gabriel Asare Okyere (PhD) Introduction to Multivariate Data Analysis May 17, 2023 18 / 41
 Descriptive Statistics

Descriptive Statistics
The sample correlation is equal to the sample covariance if
measurements are standardized.

Covariance and correlation measure linear association. Other

non-linear dependencies may exist among variables even if
rjk = 0.

A population correlation of zero means no linear association but

it does not necessarily imply independence.

The sample correlation (rij ) will vary about the value of the
population correlation (ρij )
Prof. Gabriel Asare Okyere (PhD) Introduction to Multivariate Data Analysis May 17, 2023 19 / 41
 Descriptive Statistics

Descriptive Statistics
Sums of squares and cross-products:
n
X
akk = (xik − x̄k )2 , k = 1, . . . , p
i=1
Xn
ajk = (xij − x̄j )(xik − x̄k ), k, j = 1, . . . , p
i=1

Sample statistics can be organized as vectors and matrices:

x̄ is the p × 1 vector of sample means.
S is the p × p symmetric matrix of variances (on the diagonal)
and covariances (the off-diagonal elements).
R is the p × p symmetric matrix of sample correlations.
Diagonal elements are all equal to 1.
Prof. Gabriel Asare Okyere (PhD) Introduction to Multivariate Data Analysis May 17, 2023 20 / 41
 Descriptive Statistics

Example: Bivariate Data

Data consist of n = 5 receipts from a bookstore. On each
receipt we observe the total amount of the sale (GHS) and the
number of books sold (p = 2). Then:
   
x11 x12 42 4
x21 x22  52 5
   
X5×2 = x 31 x 32
 = 88 7
  
x41 x42  58 4
x51 x52 60 5

Sample mean vector is:

   
x̄ 60
x̄ = 1 =
x̄2 5
Prof. Gabriel Asare Okyere (PhD) Introduction to Multivariate Data Analysis May 17, 2023 21 / 41
 Descriptive Statistics

Example: Bivariate Data

Sample covariance vector is:

   
s11 s12 294.0 19.0
S= =
s21 s22 19.0 1.5

Sample correlation vector is:

   
r11 r12 1 0.90476
R= =
r21 r22 0.90476 1

Prof. Gabriel Asare Okyere (PhD) Introduction to Multivariate Data Analysis May 17, 2023 22 / 41
 Distance

Outline of Presentation

1 Introduction

2 Organization of Data and Notation

3 Descriptive Statistics

4 Distance

Prof. Gabriel Asare Okyere (PhD) Introduction to Multivariate Data Analysis May 17, 2023 23 / 41
 Distance

Distance

Multivariate methods rely on distances between units.

Clustering: group units that are ’closest’ in some sense.

Classification: allocate each unit to the ’closest’ group.

Distance can be defined in different ways.

Prof. Gabriel Asare Okyere (PhD) Introduction to Multivariate Data Analysis May 17, 2023 24 / 41
 Distance

Euclidean Distance
Straight-line distance from a point P = {x1 , x2 , . . . , xp }′ in p
dimensions to the origin O is
q
d(O, P) = x12 + x22 + · · · + xp2

All points P at an equal squared distance c 2 from the origin

satisfy:
p
X
2
d (O, P) = x12 + x22 + ··· + xp2 = xj2 = c 2 ,
j=1

which defines a hypersphere centered at O.

Prof. Gabriel Asare Okyere (PhD) Introduction to Multivariate Data Analysis May 17, 2023 25 / 41
 Distance

Example

Let P = {42, 52, 48, 58}′ be a given point. Find the straight line
distance of this point from the Origin.

Solution:
Given P = {x1 , x2 , x3 , x4 }′ = {42, 52, 48, 58}′
The required distance is given by:
q
d(O, P) = x12 + x22 + x32 + x42
√
d(O, P) = 422 + 522 + 482 + 582 = 100.6777

Prof. Gabriel Asare Okyere (PhD) Introduction to Multivariate Data Analysis May 17, 2023 26 / 41
 Distance

Euclidean Distance
Euclidean distance between two arbitrary points P and Q with p
coordinates: P = (x1 , x2 , . . . , xp )′ , Q = (y1 , y2 , . . . , yp )′ is:
v
u p
uX
d(P, Q) = t (xj − yj )2
j=1

Even though p variables may be observed with different

precision, Euclidean distance gives equal weight to all.
All points P at an equal squared distance c 2 from Q satisfy:
p
X
2
d (P, Q) = (xj − yj )2 = c 2
j=1

a hypersphere centered at Q
Prof. Gabriel Asare Okyere (PhD) Introduction to Multivariate Data Analysis May 17, 2023 27 / 41
 Distance

Example
Let P = {42, 52, 48, 58}′ and Q = {4, 5, 4, 3}′ be a given points.
Find the straight line distance between P and Q.

Solution:
Given P = {42, 52, 48, 58}′ and Q = {4, 5, 4, 3}′ The required
distance is given by:
v
u p
uX
d(P, Q) = t (xj − yj )2
j=1
p
d(P, Q) = (42 − 4)2 + (52 − 5)2 + (48 − 4)2 + (58 − 3)2
d(P, Q) = 92.8116

Prof. Gabriel Asare Okyere (PhD) Introduction to Multivariate Data Analysis May 17, 2023 28 / 41
 Distance

Standardized Distance

Suppose that the variability in each of the p dimensions

(variables) is different.

We wish to give more weight in the distance calculation to those

dimensions (variables) that are measured more precisely.

Weights are inversely proportional to the standard deviation in

the measurements:
v
u p
uX  2
xj − yj
d(P, Q) = t
j=1
sj

Prof. Gabriel Asare Okyere (PhD) Introduction to Multivariate Data Analysis May 17, 2023 29 / 41
 Distance

Example
A set of paired measurements (x1 , x2 ) on two variables yields
x̄1 = x̄2 = 0, s11 = 4 and s22 = 1. Suppose the x1 measurements are
unrelated to the x2 measurements; that is, measurements within a
pair vary independently of one another. Since the sample variances
are unequal, we measure the square of the distance of an arbitrary
point P = (x1 , x2 ) to the origin O = (0, 0) by

x12 x22
d 2 (O, P) = +
4 1
All points (x1 , x2 ) that are a constant distance 1 from the origin
satisfy the equation
x12 x22
+ =1
4 1

Prof. Gabriel Asare Okyere (PhD) Introduction to Multivariate Data Analysis May 17, 2023 30 / 41
 Distance

Example cont’d

We can then calculate the standard distance between P and Q as

follows: v
u p
uX  2
xj − yj
d(P, Q) = t
j=1
sj

Prof. Gabriel Asare Okyere (PhD) Introduction to Multivariate Data Analysis May 17, 2023 31 / 41
 Distance

Standardized Distance (cont’d)

If we define P = (x1 , x2 , . . . , xp )′ , Q = (y1 , y2 , . . . , yp )′ and
D = diag sjj , then
p
d(P, Q) = (P − Q)′ D −1 (P − Q)

This measure of distance does not account for correlations

among variables: D is a diagonal matrix with all covariances set
equal to zero.

All points P at the same standardized distance c from the origin

satisfy:
(P − O)′ D −1 (P − O) = P ′ D −1 P = c 2 ,
Prof. Gabriel Asare Okyere (PhD) Introduction to Multivariate Data Analysis May 17, 2023 32 / 41
 Distance

Standardized Distance (cont’d)

Any P at a standardized distance c from Q satisfies
(P − Q)′ D −1 (P − Q) = c 2
This defines a hyper-ellipsoid centered at Q, with axes parallel to
the coordinates axes. The half-length of the axis parallel to the
√
jth coordinate axis is equal to c sjj

Prof. Gabriel Asare Okyere (PhD) Introduction to Multivariate Data Analysis May 17, 2023 33 / 41
 Distance

Other Distance Measures

Suppose now that the various measurements do not vary

independently.

What is a reasonable distance measure when the variability in

each direction is different and the variables are correlated?

Prof. Gabriel Asare Okyere (PhD) Introduction to Multivariate Data Analysis May 17, 2023 34 / 41
 Distance

Properties of Distance Measures

For any three points P = (x1 , x2 , . . . , xp )′ , Q = (y1 , y2 , . . . , yp )′ ,

and R = (z1 , z2 , . . . , zp )′ , a distance measure must satisfy:
- d(P, Q) = d(Q, P)
- d(P, Q) > 0 if P ̸= Q
- d(P, Q) = 0 if P = Q
- d(P, Q) ≤ d(P, R) + d(R, Q) triangle inequality

Prof. Gabriel Asare Okyere (PhD) Introduction to Multivariate Data Analysis May 17, 2023 35 / 41
 Distance

General Distance measure

A general distance measure is

p
d(P, Q) = (P − Q)′ A(P − Q)

where A is a symmetric positive definite matrix, a matrix with

entries ajk = akj such that the distances are always non-negative.

For p = 2,
d(P,
p Q) =
2
[(x1 − y1 )2 a11 + 2(x1 − y1 )(x2 − y2 )a12 + (x2 − y2 )2 a22 ]

Prof. Gabriel Asare Okyere (PhD) Introduction to Multivariate Data Analysis May 17, 2023 36 / 41
 Distance

General Distance (cont’d)

All points P = (x1 , x2 , . . . , xp )′ a constant distance c 2 from some

fixed point Q = (y1 , y2 , . . . , yp )′ satisfy

d 2 (P, Q) = (P − Q)′ A(P − Q) = c 2

which is the equation of an ellipse. See figure in next

transparency for the case p = 2.

The axes of the ellipse are parallel to the set of new axes (x̃1 , x̃2 )
obtained by rotating the original axes by an angle θ

Prof. Gabriel Asare Okyere (PhD) Introduction to Multivariate Data Analysis May 17, 2023 37 / 41
 Distance

General Distance (cont’d)

Figure 2:

Prof. Gabriel Asare Okyere (PhD) Introduction to Multivariate Data Analysis May 17, 2023 38 / 41
 Distance

Statistical Distance

When the measurements are correlated, we can construct a

statistical distance that accounts for correlations and unequal
variances by:
- First rotating the axes to be parallel to the axes of the ellipsoid
- Then using the expression for a standardized distance
We can re-express any point with respect to the rotated
coordinates. For P = (x1 , x2 )′ , we have

x̃1 = x1 cos(θ) + x2 sin(θ)

x̃2 = −x1 sin(θ) + x2 cos(θ).

Prof. Gabriel Asare Okyere (PhD) Introduction to Multivariate Data Analysis May 17, 2023 39 / 41
 Distance

Statistical Distance (cont’d)

Figure 3:
Prof. Gabriel Asare Okyere (PhD) Introduction to Multivariate Data Analysis May 17, 2023 40 / 41
 Distance

Statistical Distance (cont’d)

A distance measure that automatically does this is obtained by

choosing A as the inverse of the covariance matrix.

The squared statistical distance between P = (x1 , x2 , . . . , xp )′

and Q = (y1 , y2 , . . . , yp )′ is

d 2 (P, Q) = (P − Q)′ S −1 (P − Q)

This is also called the squared Mahalnobis distance

When measurements are uncorrelated this becomes the

standardized distance.
Prof. Gabriel Asare Okyere (PhD) Introduction to Multivariate Data Analysis May 17, 2023 41 / 41