Course Number AST 307
Course Title: Multivariate Statistics I
Text Book:
Applied Multivariate Statistical analysis
RICHARD A. JOHNSON & DEAN W. WICHERN, 6th edition
1
INTRODUCTION
Data analysis, while interesting with one variable, becomes truly fascinating and
challenging when several variables are involved.
Researchers in the biological, physical, and social sciences frequently collect
measurements on several variables.
The concepts of a matrix and of matrix manipulations are important. We introduce
matrices as they appear naturally in our discussions, and we then show how they
simplify the presentation of multivariate models and techniques.
The need to understand the relationships between many variables makes
multivariate analysis an inherently difficult subject.
2
More mathematics is required to derive multivariate statistical techniques for making
inferences than in a univariate setting.
It will become increasingly clear that many multivariate methods are based upon
an underlying probability model known as the multivariate normal distribution.
Multivariate analysis is a "mixed bag." It is difficult to establish a classification scheme
for multivariate techniques that both is widely accepted and indicates the
appropriateness of the techniques. One classification distinguishes techniques
designed to study interdependent relationships from those designed to study
dependent relationships. Another classifies techniques according to the number of
populations and the number of sets of variables being studied.
3
The objectives of scientific investigations to which multivariate methods most naturally
lend themselves include the following:
1. Data reduction or structural simplification. The phenomenon being studied is
represented as simply as possible without sacrificing valuable information.
It is hoped that this will make interpretation easier.
2. Sorting and grouping. Groups of "similar" objects or variables are created,
based upon measured characteristics. Alternatively, rules for classifying objects
into well-defined groups may be required.
3. Investigation of the dependence among variables. The nature of the relationships
among variables is of interest. Are all the variables mutually independent or
are one or more variables dependent on the others? If so, how?
4. Prediction. Relationships between variables must be determined for the purpose
of predicting the values of one or more variables on the basis of observations
on the other variables.
5. Hypothesis construction and testing. Specific statistical hypotheses, formulated 4
in terms of the parameters of multivariate populations, are tested.
APPLICATIONS OF MULTIVARIATE TECHNIQUES
Data reduction or simplification
•Using data on several variables related to cancer patient responses to radiotherapy,
a simple measure of patient response to radiotherapy was constructed.
• Track records from many nations were used to develop an index of performance
for both male and female athletes.
• Multispectral image data collected by a high-altitude scanner were reduced to a form
that could be viewed as images (pictures) of a shoreline in two dimensions.
• Measurements of several physiological variables were used to develop a screening
procedure that discriminates alcoholics from non-alcoholics.
• Investigation of the dependence among variables
5
• Data on several variables were used to identify factors that were responsible for
client success in hiring external consultants.
• Measurements of variables related to innovation, on the one hand, and variables
related to the business environment and business organization, on the other hand,
were used to discover why some firms are product innovators and some firms are not.
Prediction
• The associations between test scores and several high school performance variables
and several college performance variables were used to develop predictors of
success in college.
6
Hypotheses testing
• Several pollution-related variables were measured to determine whether levels for
a large metropolitan area were roughly constant through out the week, or
whether there was a noticeable difference between weekdays and weekends.
• Experimental data on several variables were used to see whether the nature of
the instructions makes any difference in perceived risks, as quantified by test scores.
.
• Data on several variables were used to determine whether different types of firms
in newly industrialized countries exhibited different patterns of innovation.
7
Multivariate Data Example
8
9
What is Multivariate Statistical Analysis?
Multivariate statistical analysis refers to multiple advanced techniques for
examining relationships among multiple variables at the same time.
Researchers use multivariate procedures in studies that involve more than
one dependent variable (also known as the outcome of interest), one or
more independent variables. Researchers often hypothesize that a given
outcomes of interest is effected or influenced by one or more predictors.
10
Examples of Multivariate Data
Units Variables
1. Students Several exam scores in a single course
2. Students Score in Statistics, Mathematics, and Physics tests
3. People Height, weight, percentage of body fat, resting heart rate
4. Companies Expenditures for advertising, labor, raw materials
5. Applicants for
bank loans Income, education level, length of residence, savings
account, current debt load
11
12
13
14
15
16
Scatterplots
Scatterplots are bivariate or trivariate plots of variables against each other. They help
us understand relationships among the variables of a data set.
17
18
Chernoff-Flury Faces
If we are interested in condensing a structure into 2D elements, we have to consider
alternative graphical techniques. The idea of using faces goes back to Chernoff (1973)
and has been further developed by Bernhard Flury. We follow the design described in
Flury and Riedwyl (1988) which uses the following characteristics.
1 right eye size 11 right upper hair line
2 right pupil size 12 right lower hair line
3 position of right pupil 13 right face line
4 right eye slant 14 darkness of right hair
5 horizontal position of right eye 15 right hair slant
6 vertical position of right eye 16 right nose line
7 curvature of right eyebrow 17 right size of mouth
8 density of right eyebrow 18 right curvature of mouth
9 horizontal position of right eyebrow 19-36 like 1-18, only for the left side.
10 vertical position of right eyebrow
19
20
DISTANCE
Most multivariate techniques are based on the simple concept of distance. So you need to familiar
with two kind of distances: (i) Straight-line (Euclidean) distance and (ii) Statistical distance.
Consider a point P=(x1, x2) in the plane. The straight-line distance d (O, P) from P to the origin O=(0,
0) is
If P has p coordinates so that P=( x1, x2, … xp), the straight line distance from O= (0, 0, …0) is
All points (x1, x2, … xp) that lie a constant squared distance, such as c2 from the origin satisfy the
equation
The straight-line distance between two points P and Q with coordinates P= (x1, x2, … xp) and Q
= (y1, y2, … yp) is given by
21
1. Find the distance between the points (2, 3) and (0, 6).
2. Find the distance between point (-1, -3) and the
midpoint of the line segment joining (2, 4) and (4, 6).
3. Find x so that the distance between the points (-2, -3) and (-3, x)
is equal to 5.
4. Find the point (0, y) that is equidistant from (4, -9) and (0, -2).
22
Straight-line (Euclidean) distance is unsatisfactory for statistical purposes.
23
24
25
26
• Euclidean distance only makes sense when all the dimensions have the same
units (like meters), since it involves adding the squared value of them.
•When you are dealing with probabilities, a lot of times the features have different
units. For example: I have a model for men and women, based on their weight [Kg]
and height [m]. I know the mean and covariance for each. Now I get a new
measurement set of weight and height and I try to decide if it's a man or a woman.
•I can use the Mahalanobis distance from the models of both men and women to
decide which is closer, meaning which is more probable.
•The Mahalnobis distance transforms the random vector into a zero mean vector
with an identity matrix for covariance. In that space, the Euclidean distance is
safely applied.
27
28
What is Ellipse and How to Graph an Ellipse
An ellipse is a set of points on a plane, creating an oval, curved shape, such that the sum of the
distances from any point on the curve to two fixed points is a constant (always the same). An ellipse
is basically a circle that has been squished either horizontally or vertically.
The point in the middle of the ellipse is called the center and is named (h, v).
The major axis is the line that runs through the center of the ellipse the long way. The variable a is
the letter used to name the distance from the center to the ellipse on the major axis. The endpoints of
the major axis are on the ellipse and are called vertices.
The minor axis is perpendicular to the major axis and runs through the center the short way. The
variable b is the letter used to name the distance to the ellipse from the center on the minor axis.
Because the major axis is always longer than the minor one, a > b. The endpoints on the minor axis
are called co-vertices.
The horizontal equation is
with the center at (h, v), major axis of 2a, and minor axis of 2b.
The vertical equation is
When the bigger number a is under x, the ellipse is
horizontal; when the bigger number is under y, it’s
29
vertical.
Center: (h, v)
Horizontal Ellipse Vertices: (h ± a, v)
Co-vertices: (h, v ± b)
Length of Major Axis: 2a
Length of Minor Axis: 2b
Center: (h, v)
Vertices: (h, v ± a)
Vertical Ellipse Co-vertices: (h ± b, v)
Length of Major Axis: 2a
Length of Minor Axis: 2b
30
Example: Find the elliptical equation for
The center is located at (h, v), or (–1, 2).
31
Example: Find the elliptical equation and draw the ellipse for
Example: Draw the ellipse for
32
33
34
In Multivariate data, four basic data types are as follows:
1. A single sample with several variables measured on each sampling unit (subject
or object);
Test the hypothesis that the means of the variables have specified values.
2. A single sample with two sets of variables measured on each sampling unit;
Find a model to predict one set of variables from the other set (multivariate
multiple regression).
3. Two samples with several variables measured on each sampling unit;
Compare the means of the variables across the two samples (Hotelling’s
T2-test).
4. Three or more samples with several variables measured on each sampling unit.
Compare the means of the variables across the groups (multivariate
analysis of variance [MAVOVA]).
35
36
37
38
39
40
41
42
43
44
45