Multivariate Linear Regression

S Ganesh, Massey University, Palmerston North, New Zealand

ã 2010 Elsevier Ltd. All rights reserved.

In the simplest case, we may be interested in study-

Glossary
ing the relationship between two sets of variables, say,
Canonical correlation analysis – A technique to Y1, Y2, . . ., Yq and X1, X2, . . ., Xp , in particular, a linear
explain or summarize the relationship between two relationship between them. This approach is the well-
sets of variables by finding a linear combination of known linear-regression modeling, where we assume
each set of variables that yields the highest possible that the variables Y ’s are dependent on the variables
correlation between the composite variable for set A X ’s. Although in a general linear modeling framework,
and the composite variable for set B. both the Y ’s and the X ’s could be quantitative
Dependent and independent variables – (or metric) and qualitative (or nonmetric), the most
Statistical models normally specify how one set of practical and commonly used case of linear-regression
variables, called dependent variables, functionally modeling assumes both the Y ’s and the X ’s to be
depend on another set of variables, called quantitative in nature.
independent variables. The term ‘(in)dependent’ In statistical terminology, Y’s are called the response
reflects only the functional relationship between (or dependent or target or output) variables and X ’s are
variables within a model. known as the regressor (or explanatory or independent or
MANOVA – Multivariate analysis of variance, a predictor or input) variables.
generalized form of analysis of variance (ANOVA), We may consider three distinguished cases, also indi-
is a technique which determines the effects of cated by Rencher (2002), according to the number of
independent variables on multiple continuous response and regressor variables:
dependent variables, and is generally used to
compare several groups with respect to multiple 1. Simple linear regression. Here, the interest is on one
continuous variables. response Y and one regressor X; for example, predict-
Method of least squares – A well-founded ing college freshman grade-point average (GPA) based
technique for estimating the unknown parameters in on the student’s SAT score.
a linear regression model, based on minimization of Usually, a model is postulated by relating the
the sum of squared differences between observed response variable to the regressor variable with un-
and modeled responses. known parameters. A model that is linear in these para-
Multivariate regression – Multivariate (linear) meters is called the simple linear regression model. This
regression is a technique aimed at finding a linear model, also referred to as a straight line model, is fitted
relationship between many response variables and via a technique known as the least squares and takes the
multiple explanatory variables. form Y ¼ b0 þ b1X, where b0 and b1 are the so-called
Regression analysis – It provides a best-fit intercept and slope parameters. The reader may read
mathematical equation for the relationship between any intermediate-level statistics textbook for further
the response variable(s) and regressor variable(s) or details (e.g., Ott and Longnecker, 2001).
covariates. 2. Multiple linear regression. Multiple regression analysis is
Wilks’ lambda – A general test statistic used in used whenever we wish to model the relationship be-
multivariate tests (e.g., MANOVA) based on a tween one response variable and more than one regres-
combination of dependent variables. Several other sor variable. In the preceding example, we could
statistics are special cases of Wilks’ lambda. attempt to improve our prediction of college GPA by
using, for example, high school GPA, Scholastic Apti-
tude Test (SAT) scores, and rating of school.
Many different forms of relationship are possible,
In many scientific studies, numerous variables (or mea- but the overwhelming emphasis in practical applica-
surements) are obtained for each individual or unit studied. tions is on the linear relationship Y ¼ b0 þ b1X1 þ
Regression analysis is a statistical methodology for predict- b2X2 þ . . . þ bpXp , known as the multiple linear
ing the value of one (or more variables) from a collection of regression model. The regression coefficients b0,
other variables. Regression models have been used exten- b1, . . ., bp are model parameters whose values need to
sively in the education literature (see, e.g., Hsu, 2005). be estimated from the given data.

324
 Multivariate Linear Regression 325

3. Multivariate (multiple) linear regression. Multivariate mul- researcher who then observes the changes in the yield
tiple regression analysis arises when we have more than and nutritional responses. We shall interpret the regres-
one response variable, and we wish to model the rela- sion model conditional on the type of regressor variables.
tionship between these variables and a set of regressor A detailed account of modeling when the regressors are
variables. Attention is again focused almost exclusively fixed and when they are random, is given by Rencher (2002).
on linear relationships. In the preceding example, we In the univariate case, the linear regression model is
may wish to predict freshman college GPA (in the defined by the following equation:
sciences, arts, and humanities) as well as the number
yk ¼ b0 þ b1 Xk1 þ b2 Xk2 þ . . . þ bp Xkp þ ek ½1
of years of college the student will complete based on
high school GPA, SAT, and rating of school. Here, yk is the response for the kth individual in the
Note that, in multivariate multiple linear regression, given sample, Xkj is the value of the jth regressor variable
multivariate refers to the response variables and multi- for the kth individual, and ek is the random departure
ple represents the independent variables so that (2) term for that individual representing the measurement
above can be regarded as univariate linear regression. errors. Since random sampling is assumed, the ek’s are
The focus of this article is on multivariate linear re- taken to be independent and identically distributed
gression and the main aim is to give a detailed explo- random variables, each having mean zero and constant
ration of the topic including examples. variance s2.
A mathematical formulation of the multiple linear If we collect together all the yk , bj , and ek into the
regression model is provided in this article, with relation- vectors (with superscript T denoting matrix transposi-
ship to univariate (multiple) linear regression and the tion) y ¼ (y1 y2 . . . yn)T, b ¼ (b0 b1 . . . bp)T, and « ¼
method of least squares. The underlying assumptions are (e1 e2 . . . en)T, and write the regressor variable values in
discussed and model diagnostics carried out. This article the n  (p þ 1) regressor matrix
0 1
also covers testing of hypotheses, ANOVA partitioning, 1 X11 ... X1p
and simultaneous confidence intervals. The application of B .. .. .. C
X¼ B
@1 . .
C
. A
the fitted models is illustrated via an example and suitable
computer software. 1 Xn1 ... Xnp
Even though most of the basic theory behind the topic
then, the above model [1] can be written in the compact
of interest, multivariate linear regression, is generally form
available in many textbooks and research articles, the
following account is based on the seminal work of Khattree y ¼ Xb þ « ½2
and Naik (1995) and Krzanowski (1998).
In this formulation, « is a random vector whose mean is
0 (n  1 vector of zeroes) and whose dispersion (or
covariance matrix) is s2lp, lp being the p  p identity
Formulation of the Multivariate Linear matrix, implying that the ek ’s are identically and inde-
Regression Model pendently distributed with mean 0 and variance s2.
In the multivariate situation, that is, when there are
Suppose that we observe, say, q response variables Y1, . . ., more than one response variable, we have to allow each
Yq and p regressor variables X1, . . . , Xp on, say, n sample Yi to have its own linear relationship with all the Xj ’s.
individuals or units. Here, we shall focus on the behavior Thus, we have to specify q different linear models, one
of response variables in terms of the regressor variables. between each Yi and the set of Xj . By analogy with the
The regressor variables are usually assumed to be measured above eqn [2], we can write these models as
without error, but the response variables are subject to
measurement errors. yi ¼ Xbi þ «i ½3
Furthermore, the regressor variables may be fixed or
random. In the above examples, all X ’s are random vari- where yi ¼ ( yi1 yi2 . . . y in)T, bi ¼ (bi0 bi1 . . . bip)T, and «i
ables in that a student is selected at random and all the Y ’s ¼ (ei1 ei2 . . . ein)T, for i ¼ 1, . . ., q.
and X’s are measured for that individual (so, the variables Here, yik is the value observed for the ith dependent
are not under the control of the researcher). In some cases, variable on the kth sample unit, eik is the error term
especially in experimental situations, the X ’s are fixed so corresponding to the ith dependent variable and kth indi-
that they are under the control of the researcher. For vidual, ( p þ 1)  1 vector bi are the parameters
example, we may wish to relate, say, yield and the nutri- corresponding to the ith dependent variable, and X is
tional value of corn to various levels of a certain fertilizer the same matrix as before.
and several varieties. Here, the amounts of fertilizer to be In fact, the q equations in [3] can be represented
applied and the choices of variety are controlled by the compactly via the following matrix notation:
326 Statistics

Y ¼ XB þ j ½4 (and products) via the well-known least-squares approach.

To demonstrate, we wish to minimize,
where the q vectors yi are put side by side as columns into
the n  q matrix Y, say, the response matrix, the q vectors X
q X
q
«T
i «i ¼ ðyi  Xbi ÞT ðyi  Xbi Þ
bi similarly into the (p þ 1)  q matrix B as columns, say, i¼1 i¼1
the parameter matrix, and the q column vectors «i into the
n  q matrix j, say, the error matrix. where yi, bj , and «i are as defined before. This is equiva-
It is possible to treat each of the q equations in [3] lent to minimizing the trace (i.e., sum of diagonal ele-
independently of each other as a univariate linear regres- ments) of the p  p matrix (Y  XB)T (Y  XB), which
sion. However, since the response variables may them- results in a set of normal equations,
selves be correlated with each other, this dependence ðXT XÞB ¼ XT Y
should also be taken into account when fitting the multi-
Solving these equations yield the estimator B, ^ given
variate linear regression model. This, indeed, suggests the
need for using the model in eqn [4], where all the depen- by eqn [5], as the least squares estimator of parameter
dent variables are analyzed together. Even though the matrix B.
regression parameter estimates and their standard errors Note that, as in the univariate linear regression case,
from the multivariate approach should be identical to we assume that the regressor matrix X is of full rank, that
those obtained from the set of univariate models, the is, rank (X) ¼ p þ 1. This implies the existence of (XTX)1
added advantage with the multivariate regression model as rank (XTX) ¼ rank(X). Although a generalized inverse
is that it allows for joint inferences involving two or more of (XTX) may be used when X is not of full rank, extra care
of the univariate models. For example, one may test the is needed when interpreting the regression results in such
overall significance of the regression parameters (across situations.
Moreover, since we may write B ^0 b
^ ¼ [b ^q], using
^1 . . . b
all fitted univariate models) using a single statistical test.
Furthermore, one may also test whether the coefficients appropriate columns from eqn [5],
of a particular regressor are the same across all fitted ^ i ¼ ðXT XÞ1 XT y for i ¼ 1; . . . ; q
b ½6
i
(univariate) models. Such hypotheses tests are explained
later in this article. ^ is an unbiased estimator of B,
It is easy to observe that B
Once again, the n sample individuals are assumed ^
that is, E(B)¼B. Furthermore, we can show
to be independent, but now there is an association ^ i; b
covðb ^ j Þ ¼ sij ðXT XÞ1 fori; j ¼ 1; . . . ; q ½7
among the error terms ek1, . . ., ekq (k ¼ 1, . . ., n)
corresponding to the same individual. Consequently, we Equation [6] appears to imply that the regression coeffi-
can treat the rows of the error matrix j as independent cients in the multivariate linear regression model have the
observations from a distribution with mean vector zero and same estimates as they would if each response variable
dispersion matrix, say, S, and for most practical purposes, was regressed separately on the entire set of regressor
this distribution is assumed to be multivariate normal. variables, that is, when fitting univariate linear regression
In other words, while the ith column of j has the covariance models. However, this may not be as simple as it sounds
matrix, say, s2iiln, ln being the due to the fact that all the individual b ^ ij in B
^ are inter-
Pn  n identity matrix, a typical
row of j has that as, say, ¼ [sij], a symmetric q  q correlated in the case of multivariate regression modeling.
matrix that is assumed to be positive definite, where given X, In other words, those b ^ ij within a column of B ^ are corre-
cov(Yi, Yj) ¼ cov(«i, «j) ¼ sijln, for i, j ¼ 1, . . ., q. The lated because of possible correlations among the regressor
assumption of multivariate normality for error vectors is variables, and those in different columns of B ^ because of
needed for hypothesis testing and construction of confi- the correlations among the response variables.
dence regions, even though it is not needed for estimating Hence, instead of performing statistical inference
parameters in the multivariate regression modeling. (hypothesis tests, etc.) separately on each bi, that is, in
a univariate regression scenario for response variable
Yi, we should perform multivariate tests of hypotheses,
Fitting the Multivariate Linear Regression etc., about the regressor matrix B. This, in fact, would
Model via Least-Squares Estimation allow testing of hypotheses with respect to parameters
within univariate models as well as between such models.
Maximizing the likelihood for the model in eqn [4] under
the normality condition produces an estimator of B given by
^ ¼ ðXT XÞ1 XT Y
B ½5 Statistical Inference

The same estimator results from the simple process The hypothesis tests associated with the regression para-
of minimizing the criterion error sum of squares meters in B are the analogs in multivariate analysis of
 Multivariate Linear Regression 327

variance (MANOVA) of the analysis of variance (ANOVA) where 0 denotes the matrix of zeroes, and, say, L is a matrix
F-tests for univariate linear (multiple) regression. The gen- of size r  ( p þ 1), and M is a q  s matrix. L may be
eral theory behind MANOVA is beyond the scope of this regarded as defining a linear function on the regressor side
article, but can be found elsewhere in the encyclopedia and and M as defining a linear function on the response side.
in many textbooks on multivariate analysis, for example, The L and M matrices play different roles. So, they
Krzanowski (2000). need to be chosen carefully: L is used to obtain a linear
Here, we shall simply specify the relevant matrices function of the regression parameters within the models,
from which the test statistics are obtained, and by analogy thus, leading to, for example, testing the significance of
with univariate linear regression. The total variation asso- a single regressor or difference between two regressors
ciated with the response variables is given by the p  p associated with each of the response variables; M, on the
matrix of (mean-corrected) total sums of squares and other hand, is used for testing the significance of a lin-
products (SSP) defined as ear function of regression parameters associated with
different response variables but corresponding to the
T ¼ YT Y  nyyT ½8
same set of regressors, for example, testing the difference
where y  ¼ [
y1, y
2 . . . y
q] is the matrix of sample mean
T
between the coefficients of, say, X1 in the models for
vectors of the yi ¼ (yi1 yi2 . . . yin)T, i ¼ 1, . . ., q. Note here Y1 and Y2. Hence, using both L and M would lead to
that the ijth element of T corresponds to the sample interesting inferences.
covariance between the ith and jth response variables When we wish to test only the hypotheses associated
(i.e., the mean-corrected SSP divided by appropriate with regression parameters, M is set as the q  q identity
degrees of freedom). matrix. Similarly, when the significance of the entire
Assuming that the regressor matrix X is of full-rank multivariate regression model fitted is tested, matrix L is
p þ 1, the total matrix T can be partitioned into the also set as an identity matrix of size (p þ 1)  (p þ 1).
hypothesis matrix, say, H for testing the overall signifi- The corresponding hypothesis and error matrices are
cance of regression and the error or residual matrix, say, E. then given by
The H and E matrices, in the expression T ¼ H þ E, are ^
^ T LT ½LðXT XÞ1 LT LBM1
H ¼ MT B ½11
given by
^ T XT Y  nyyT
H¼B ½9
and
^ T XT YÞM
E ¼ MT ðYT Y  B ½12
and
^ T XT Y ¼ ðY  XBÞ
^ T ðY  XBÞ
^ There are many test statistics available to test the hypo-
E ¼ YT Y  B ½10
theses stated above, most being based on the functions of
Note that, while Y ^ ¼ BX^ is the matrix of the predicted eigenvalues, say, l1, l2, . . ., lq , of the matrix HE1,
values, the diagonal elements of the matrices T, H, and E, assuming E1 exists. These include:
respectively, represent the usual total, regression and error
1. Roy’s maximum root: The largest eigenvalue, say, lmax
sums of squares for the corresponding dependent variables
2. Hotelling-Lawley trace:
in the univariate linear regression scenario. Furthermore,
q
X
eqn [10] implies that an unbiased estimate of S is given by
li ¼ TraceðHE1 Þ
E/(np1). i¼1

3. Pillai’s trace:
X
q
li
Testing of (Linear) Hypotheses: ¼ TraceðHðH þ EÞ1 Þ
i¼1
1 þ li
Multivariate Tests

In the context of multivariate linear regression models, the 4. Wilks’ lambda:

hypotheses examined may be, in general, associated with Y
q
1 jEj
the functions of the regressor matrix B. Here, we aim to test ¼ :
i¼1
1 þ l1 jH þ Ej
the redundancy of certain regressor variables, or use the
tests in connection with model-reduction schemes as in the (see Krzanowski (2000) or Khattree and Naik (1995) for
process of selecting regressor variables, or when a compari- further details.)
son of two or more populations is needed (where the Note that, Pillai’s trace may be regarded as a measure
regressor variables are regarded as experimental factors). of response variable variation explained by the fitted
The most popular general form of such (linear) multivariate linear regression model, and as an analog to
hypotheses is the well-known coefficient of determination (R2) in the
H0: LBM ¼ 0 vs H1: LBM 6¼ 0 context of univariate linear regression.
328 Statistics

Although many comparisons of these statistics have found in Guber (1999). It is quite natural to think of
been considered in the literature, the results are still linear modeling to predict verbal and math SAT scores
indecisive. However, the Wilks’ lambda statistic (usually for a set of values of COST, RATIO, SALARY, and
denoted by L) has been chosen by many researchers, ELIGIBLE, even though these regressor variables were
partly because of its ease of computation, but mainly for not fixed by the investigator.
the existence of distributional approximations which The main aim here is to fit a multivariate linear regres-
enable critical values to be readily found. Hence, the sion model expressing response variables VERBAL and
remainder of this article considers only the Wilks’ lambda MATH as linear functions of regressor variables COST,
test statistics. RATIO, SALARY, and ELIGIBLE, and to carry out some
Although tables of critical values for this and other hypothesis testing related to the regression parameters as
test statistics are available, we shall explore the popular well as the response variables.
F-statistic approximation for Wilks’ lambda given by Recall that, in matrix notation, the model is given by
!  Y ¼ XB þ j, where we may define matrices Y, X, B, and j as
1  L1=t st  2u Y ¼ [yVERBAL yMATH]502, X ¼ [1 xCOST xRATIO
F¼ ½13
L1=t ab xSALARY xELIGIBLE]505, and x ¼ [eVERBAL eMATH]502,
with yVERBAL, xCOST etc. being the response and regressor
where
vectors of size 50  1 (50 representing the US states),
b a þ1 1 being a 50  1 vector of 1s and the 50  1 vectors
s ¼ ðn  p  1Þ ; a ¼ rankðLÞ; b ¼ rankðH þ EÞ;
2 «VERBAL and «MATH representing the error respective
u ¼ ðab  2Þ=4 errors; and
B ¼ [bVERBAL bMATH]52 with, say, bVERBAL ¼ (bV_Int
and bV_COST bV_RATIO bV_SALARY bV_ELIGIBLE)T and bMATH ¼
8 pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi (bM_Int bM_COST bM_RATIO bM_SALARY bM_ELIGIBLE)T repre-
< ða2 b 2 4Þ=ða2 þ b 2 5Þ if ða2 þ b 2 5Þ > 0
senting the regression parameters.
t¼
: Suppose we wish to test the following hypotheses:
1 otherwise
 H01: significance (overall) of the entire model;
The F-statistics in eqn [13] is said to approximately follow  H02: significance of each of the regressor variables;
an F-distribution with degrees of freedom ab and (st – 2u).  H03: coefficients or parameter estimates associated with
The distribution is exact if min(a,b)2 (see Rao (1973) for regressors COST and SALARY are the same for each of
details). the response variables VERBAL and MATH (or are the
Furthermore, if in particular H0: LB ¼ 0 (with M ¼ Iq) same in the two equations given by eqn [3]).
is rejected, then we may attempt to provide further infer-
ence on the regression parameters. For example, we may We may rewrite the above hypotheses in the form H0:
compute confidence intervals for the individual compo- LBM ¼ 0 with H01: B ¼ 0, where M ¼ I22 and L ¼ I55
nents of LB or just B (if L ¼ Ipþ1). Here, we use Roy’s H02: LBM ¼ 0, where M ¼ I22 and L ¼ (0 1 0 0 0)15
largest eigenvalue statistic in conjunction with the cor- for testing the significance of regressor variable COST,
responding (approximate) F-distribution (see Khattree L ¼ (0 0 1 0 0)15 for testing the significance RATIO, etc.
   
1 01000
and Naik (1995) for further details). H03: LBM ¼ 0, where M and L ¼
1 21 0 0 0 1 0 25
or equivalently, test bV COST  bM COST ¼ 0 and bV SALARY 
bM SALARY ¼ 0 simultaneously.
Example and Software
Statistical computing software packages, such as SAS
To illustrate the basic theory and ideas above, we shall (2002–03) and R (2008), may be used readily to carry out a
consider the following example (hereafter referred to as multivariate linear regression analysis and test specific
the SAT data): hypotheses such as those mentioned above. We shall uti-
DASL (see Web resources) describes data from lize the SAS system via coding to perform the required
1994–95 for each of the 50 US states, for the variables analyses. For a basic introduction to using SAS for statis-
current expenditure per pupil in thousands of dollars tical analyses, see Der and Everitt (2002).
(COST), the average pupil/teacher ratio (RATIO), the The annotated SAS codes are presented in Table 1. The
estimated average annual salary of teachers in thousands well-known regression procedure PROC REG of SAS has
of dollars (SALARY), the percentage of all eligible stu- been used here with additional statement MTEST for
dents taking the SAT (ELIGIBLE), then average verbal carrying out multivariate tests (e.g., Wilks’ lambda with
SAT score (VERBAL), the average math SAT score F-approximation). Though PROC REG also produces the
(MATH), and the average total score on the SAT regression parameter estimates automatically, for those
(TOTAL). Additional information on this data can be interested in using matrix manipulations for the various
 Multivariate Linear Regression 329

Table 1 SAS codes for multiple linear regression of SAT data restrictions (i.e., just estimating the regression parameters
/* read SATdata (in space-delimited text format) using eqn [5]), the estimated regression relationships to be
from external file [Link] */ VERBAL ¼ 509.699 þ 1.307  COST – 2.081  RATIO þ
data SATdata; 0.627  SALARY – 1.337  ELIGIBLE.
format STATE $13.;
MATH ¼ 536.272 þ 3.156  COST – 1.543  RATIO þ
infile ’D:\Research\MultRegress\[Link]’
firstobs=2;
1.008  SALARY – 1.567  ELIGIBLE.
input SID STATE $ COST RATIO SALARY ELIGIBLE VERBAL We can see that the regression coefficients obtained here
MATH TOTAL; are exactly the same as those obtained via matrix manip-
run; ulations shown above. The significance of these regres-
/* use matrix operations via PROC IML to compute reg sion coefficients, given by the corresponding p values in
coefs */
%let var1 = VERBAL MATH;
Table 2, are based on the univariate regression approach.
%let var2 = COST RATIO SALARY ELIGIBLE; So, they need to be interpreted with caution! Ideally, the
proc iml; /* start of IML */ significance of each of the regression parameters should be
use SATdata; examined in a multivariate sense, that is, overall with
read all var {&var1} into Y[colname=vars];
respect to the two response variables, and these results,
read all var {&var2} into X[colname=vars];
n=nrow(Y); q=ncol(Y);
obtained via PROC REG codes, are shown below.
ones=J(n,1,1); X=ones||X; p=ncol(X); Table 2 also shows the estimated error SSP matrix E
B = (inv(t(X)*X))*t(X)*Y; and the regression or hypothesis SSP matrix H (given by
print B; eqns [9] and [10]) associated with the null hypothesis H01,
quit; that is, for testing the significance of the entire fitted model.
/* use PROC REG procedure with MTEST statement */
proc reg data=SATdata;
Interpretation of these matrices is left to the reader’s discre-
model VERBAL MATH = COST RATIO SALARY ELIGIBLE; tion. The corresponding Wilks’ lambda-based multivariate
/* test significance of overall model */ test (labeled overall model) indicates that the approximate
OverallModel: mtest intercept,COST,RATIO,SALARY, F-statistic with 10 and 88 degrees of freedom has a very
ELIGIBLE / print; large value of 291.14 and a very small p value of <0.0001,
/* error and hypothesis matrices printed */
/* test significance of each regression parameter */
thus, providing very strong evidence for significance of the
COST: mtest COST; /* error and hypothesis matrices entire multivariate linear regression model fitted.
printed */ The multivariate tests (labeled COST, RATIO, SALARY,
RATIO: mtest RATIO; and ELIGIBLE) for testing the significance of each regres-
SALARY: mtest SALARY;
sor variable simultaneously among all the response variables
ELIGIBLE: mtest ELIGIBLE;
/* test coeffs of COST and SALARY are the same in
indicate that, only the variable ELIGIBLE has a very signif-
VERBAL and MATH */ icant effect on the variables VERBAL and MATH. This is
SameCoef: mtest VERBAL-MATH, COST, SALARY; indicated by the F-statistic (78.20) for the corresponding
run; Wilks’ lambda with very small p-value (<0.0001). The
quit; F-statistics for each regressor has the same degrees of free-
(Further details are available from the author, in particular, on the dom, and the other regressors COST, SALARY, and RATIO
use of PROC IML.) all appear to not significantly contribute to VERBAL and
MATH.
Finally, consider testing of the hypothesis that the
computations, SAS provides the powerful PROC IML pro- parameter estimates associated with regressors COST and
cedure. As an example, PROC IML codes for comput- SALARY are the same for each of the response variables
ing the regression parameter estimates given by B ^ in VERBAL and MATH. The corresponding F-statistic with
eqn [5], is also shown in Table 1. The estimates are given 2 and 45 degrees of freedom derived from Wilks’ lambda has
^ b
by B¼[ ^ VERBAL b ^ MATH]52, with bVERBAL ¼ (509.699 a value 3.47 and a p-value of 0.0348 (see multivariate test
1.307 2.081 0.630 1.337)T and bMATH ¼ (536.272 3.156  labeled SameCoef). Thus, at best, the hypothesis could only
1.543 1.008 1.567)T. be rejected with weak evidence, indicating that COST and
Table 2 shows the relevant portions of the edited SALARY have approximately the same coefficients in the
output obtained by running the PROC REG codes in linear regression models for VERBAL and MATH.
Table 1. We noted earlier that the regression coefficients Note here that, the advantage of fitting a multivari-
in the multivariate linear regression model have the same ate regression model (or bivariate regression model in
estimates as they would if each response variable was the above example) over fitting separate (or two, in the
regressed separately on the entire set of regressor vari- example) univariate regression models is that, in general,
ables (in univariate linear regression models), as sup- the former takes into account the correlation(s) between
ported by eqn [6]. The first part of Table 2 shows these the response variables while the latter ignores such interde-
univariate regression results indicating that with no pendency. To illustrate, consider a hypothesis testing similar
330 Statistics

Table 2 Selected and edited SAS output for multiple linear regression of SAT data
Univariate Tests:
Dependent Variable: VERBAL
Parameter Estimates
Variable DF Parameter Estimate Standard t Value Pr > |t|
Error
Intercept 1 509.69910 24.55395 20.76 <.0001
COST 1 1.30662 4.89805 0.27 0.7909
RATIO 1 2.08140 1.49332 1.39 0.1702
SALARY 1 0.62996 1.10869 0.57 0.5727
ELIGIBLE 1 1.33733 0.10740 12.45 <.0001
Dependent Variable: MATH
Parameter Estimates
Variable DF Parameter Estimate Standard t Value Pr > |t|
Error
Intercept 1 536.27243 30.22140 17.74 <.0001
COST 1 3.15598 6.02860 0.52 0.6032
RATIO 1 1.54283 1.83800 0.84 0.4057
SALARY 1 1.00796 1.36460 0.74 0.4640
ELIGIBLE 1 1.56715 0.13219 11.86 <. 0001
Multivariate Test:
Error Matrix (E) Hypothesis Matrix (H)
10379.79661 11009.830615 10499099.203 11685411.169
11009.830615 15724.438311 11685411.169 13006334.562

Multivariate Test: Overall Model Multivariate Statistics and F Approximations

Statistic Value F Value Num DF Den DF Pr > F
Wilks’ Lambda 0.0008608 291.14 10 88 <.0001
Multivariate Test: COST Multivariate Statistics and F Approximations
Statistic Value F Value Num DF Den DF Pr > F
Wilks’ Lambda 0.99105479 0.20 2 44 0.8206
Multivariate Test: RATIO Multivariate Statistics and F Approximations
Statistic Value F Value Num DF Den DF Pr > F
Wilks’ Lambda 0.94834102 1.20 2 44 0.3113
Multivariate Test: SALARY Multivariate Statistics and F Approximations
Statistic Value F Value Num DF Den DF Pr > F
Wilks’ Lambda 0.98762687 0.28 2 44 0.7604
Multivariate Test: ELIGIBLE Multivariate Statistics and F Approximations
Statistic Value F Value Num DF Den DF Pr > F
Wilks’ Lambda 0.21956036 78.20 2 44 <.0001
Multivariate Test: SameCoef Multivariate Statistics and Exact F Statistics
Statistic Value F Value Num DF Den DF Pr > F
Wilks’ Lambda 0.86648426 3.47 2 45 0.0398

to the last one above. Here, suppose we wish to compare Summary

the coefficients of regressor variable COST for the two
regression models fitted, one for response VERBAL and Linear regression models, in general, are among the most
the other for MATH. We may construct a test statistic commonly used statistical methods, while multivariate
synonymous with a two-sample t-test when using two sepa- regression models extend the basic idea to many response
rate univariate models, assuming independence between variables. The theory behind multivariate linear regression
the response variables. On the other hand, using a bivari- modeling is highly developed and easily applied to real
ate/multivariate regression model would yield us a testing problems. Implementation or fitting of multivariate linear
process synonymous with the so-called paired t-test. How- regression models is more difficult than fitting a univariate
ever, we know that, when the assumption of independence is linear regression model, but procedures are readily avail-
in question, the paired t-test is more powerful than the two- able in statistics software packages such as SAS and R.
sample t-test. Hence, it is logical to conclude that multivar- This article provided an easily digestible theory on
iate regression modeling is more powerful than fitting sepa- multivariate linear regression modeling illustrated by an
rate univariate regression models when the correlation easily understood example. It also focused on modeling
structure between the response variables are considered to with quantitative response and regressor variables. The
be an important aspect of the modeling process. analysis of models with qualitative regressor variables
 Multivariate Linear Regression 331

with quantitative responses is usually covered in a MAN- R (2008). The R foundation for statistical computing. Version 2.7.1.
[Link] (accessed October 2009).
OVA scenario with experimental factors. The case with Rao, C. R (1973). Linear Statistical Inference. New York: John Wiley &
qualitative responses and quantitative regressors could be Sons, Inc.
tackled via the typical discrimination and classification Rencher, A. C. (2002). Methods of Multivariate Analysis, 2nd edn. New
York: Wiley.
scenarios, but the case where both the response and regres- SAS (2002–2003). SAS Institute Inc. Version 9.1. Cary, NC, USA. http://
sor variables are qualitative, may be handled by techniques [Link] (accessed October 2009).
such as canonical correspondence analysis. Many of the
above scenarios are also covered in the area of generalized
linear models. Another technique that has a very close
relationship with multivariate linear regression is known Further Reading
as canonical correlation analysis (CCA). CCA provides a
symmetric relationship between the response and regres- Altunkaynak, B. and Ekni, M. (2002). Detection of influential observation
vectors for multivariate linear regression. Hacettepe Journal of
sor variables and attempts to extract most of the associa- Mathematics and Statistics 31, 139–151.
tion between the two sets of variables via the linear Goldwasser, M. A. and Fitzmaurice, G. M. (2001). Multivariate linear
functions of the two sets (Krzanowski, 2000). regression analysis of childhood psychopathology using multiple
informant data. International Journal of Methods in Psychiatric
The reader is referred to Kharttree and Naik (1995) Research 10, 1–10.
and Rencher (2002) for an extended coverage on multivar- Jobson, J. D. (1992). Applied Multivariate Data Analysis: Volume II:
iate linear regression modeling, including model diagnos- Categorical and Multivariate Methods. New York: Springer.
Johnson, R. A. and Wichern, D. W. (2007). Applied Multivariate
tics, response surface modeling, simultaneous confidence Statistical Analysis, 6th edn. Upper Saddle River, NJ: Pearson
intervals, handling fixed and random regressor variables, Prentice Hall.
and dealing with nonsingular matrices leading to the use of Kim, K. and Timm, N. (2006). Univariate and Multivariate General Linear
Models Theory and Applications with SAS, 2nd edn. New York: CRC
generalized inverses. Press.
Lutz, J. G. and Eckert, T. L (1994). The relationship between canonical
correlation analysis and multivariate multiple regression. Educational
and Psychological Measurement 54(3), 666–675.
Bibliography Weisberg, S. (2005). Applied Linear Regression, 3rd edn. Hoboken, NJ:
Wiley-Interscience.
Der, G. and Everitt, B. S. (2002). A Handbook of Statistical Analyses Wikipedia contributors (2008). Multivariate linear regression.
Using SAS, 2nd edn. London, New York: Chapman & Hall/CRC. In Wikipedia, The Free Encyclopedia (see topics such as General
Guber, D. L. (1999). Getting what you pay for: The debate over equity in Linear Model and Bayesian Multivariate Linear regression). [Link]
public school expenditures. Journal of Statistics Education 7(2), [Link]/wiki/Special:Search?search=Multivariate+Linear+
[Link] Regression(accessed October 2009).
[Link] (accessed October 2009).
Hsu, T. -C (2005). Research methods and data analysis procedures
used by educational researchers. International Journal of Research &
Methods in Education 28(2), 109–133. Relevant Websites
Kharttree, K. and Naik, D. N. (1995). Applied Multivariate Statistics with
SAS Software. Cary, NC: SAS Institute.
Krzanowski, W. J. (1998). Multivariate multiple regression. In Armitage, [Link] – DASL, The Data and
P. and Colton, T. (eds.) Encyclopedia of Biostatistics, pp 8. Story Library.
Chichester: Wiley. [Link] – SAS, superior software that gives you the power
Krzanowski, W. J. (2000). Principles of Multivariate Analysis: A User’s to know.
Perspective. Oxford (Oxfordshire), New York: Oxford University [Link] – Software download: multivariate
Press. regression Popular Software.
Ott, R. L. and Longnecker, M. T. (2001). Introduction to Statistical [Link] – The R Project for Statistical Computing.
Methods and Data Analysis, 5th edn. United States: Thomson [Link] – UCLA Department of Statistics.
Learning, Duxbury Press. [Link] – UCLA SAS Library.