§ Descriptive Statistics

◎Summation form:
1 n
⊙Sample mean: xi  i   xij , i=1, 2, …, p.
n j 1

○Sample mean: i   x p ( x ); 
all x i
i i i

xi f i ( xi )dxi , i=1, 2, …, p.

○Sample mean:  i  E ( xi ) , i=1, 2, …, p.

1 n
⊙Sample variance: sii   ( xij  xi )2 , i=1, 2, …, p.
n j 1

○Sample variance: sii   (x   )
all xi
i i
2
pi ( xi );  ( xi  i ) 2 f i ( xi )dxi , i=1, 2,…, p.


○Sample variance: sii  E ( xi  i ) 2 , i=1, 2, …, p.

1 n
○Sample standard deviation:  ii  sii  
n j 1
( xij  xi ) 2 , i=1, 2, …, p.

1 n
⊙Sample covariance: sik   ( xij  xi )( xkj  xk ) , i=1, 2, …, p; k=1, 2, …, p.
n j 1

○Sample covariance: sik    (x

all x i all x k
i  i )( xk   k ) pik ( xi , xk ) , i=1, 2,…, p; k=1, 2, …, p.

 
○Sample covariance: sik    ( xi  i )( xk   k ) dxi dxk , i=1, 2,…, p ; k=1, 2, …, p.
 

○Sample covariance: sik  E ( xi  i )( xk   k ) , i=1, 2, …, p; k=1, 2, …, p..

n

sik
 (x
j 1
ij  xi )( xkj  xk )
⊙Sample correlation coefficient: rik   , i=1, 2, …, p;
n n
sii skk
 (x
j 1
ij  xi ) 2
 (x
j 1
kj  xk ) 2

k=1, 2, …, p.

( xij  xi ) ( xij  xmin )

⊙Standardized value: ; Normalized value:
sii ( xmax  xmin )

◎Array form:
 x1 
 
x
⊙Sample means: X   2  , μ  E ( X )

 
 x p 

1
  s11 s12  s1 p 
s s22  s2 p 
⊙Sample variance and covariance: S p   ;   E ( X   )( X   )'
21

     
 
 s p1 sp2  s pp 
 1 r12  r1 p 
r 1  r2 p 
⊙Sample correlation: R   21

    
 
 rp1 rp 2  1 

◎Distance concepts
⊙Euclidean distance: d ( P, Q )  ( x1  y1 ) 2  ( x2  y2 ) 2  ...  ( x p  y p ) 2

( x1  y1 ) 2 ( x2  y2 ) 2 ( x p  y p )2
⊙Statistical distance: d ( P, Q )    ... 
s11 s22 s pp
⊙If s11=s22=…=spp, then Euclidean distance = Statistical distance.

◎Some Basics of Vector Algebra

 x1 
x 
⊙Vector X   
2


 
 x p  p 1
⊙Transpose operator: X’= [x1, x2,…, xp] 1 p
 x1   y1   x1  y1 
x   y   x  y 
⊙Addition operator: X  Y        
2 2 2 2

    
     
 x p   y p   x p  y p 
 cx1 
 cx 
⊙Multiplication by a scalar c: cX   
2

  
 
cx p 
⊙Length of X : LX  x12  x22  ...  xn2 ; LX  X ' X
⊙Length of cX= c LX  c x12  x22  ...  xn2 .
X 'Y
⊙ cos( )  . If cos( )  0 (i.e. X’Y =0), then X and Y are perpendicular.
LX LY
⊙X1, X2, …, Xk are said to be linearly dependent if there exist constants c1, c2,…, ck not all zero,
such that c1X1+ c2X2 +…+ ckXk =0.
X 'Y X 'Y 1
⊙Projection (shadow) of a vector X on a vector Y: Y Y.
Y 'Y LY LY

2
 X 'Y X 'Y
⊙Length of projection:  LX  LX cos( ) .
LY LX LY
◎Some Basics of Matrix Algebra
⊙Matrix product AB
⊙If AB=BA=I, then B is called the inverse of A and is denoted by A-1.
⊙Orthogonal matrices: Q’Q = QQ’ = I or Q’=Q-1.
⊙Let A be a ( k  k ) square symmetric matrix:
○A has k pairs of eigenvalues and eigenvectors: 1 , e1 , 2 , e2 ,..., k , ek . The eigenvectors can
be chosen to satisfy: 1= e1’e1 = e2’e2 = … = ek’ek.
○ 1 , 2 ,..., k satisfy A  I  0 .
○Spectral decomposition: A = 1e1e1 ' 2e2e2 '...  k ek ek ' and Aei  i ei .
○If X ' AX  0 , then A is said to be nonnegative definite. If equality holds only for X’= [0,
0, …,0], then A is said to be positive definite.
○If all i  0 for all i, then A is said to be nonnegative definite.
1 0  0
k 0 2  0 
○ A   i ei ei '  PP' , where P=[e1, e2, …, ek] and    .
i 1
   
 
 0 0   p 
k
1
○ A1  P1P '   ei ei ' .
i 1 i
k
○A square-root matrix: A1 / 2  P1 / 2 P'   i ei ei ' , which has following properties:
i 1
‧(A1/2)’ =A1/2. (That is A1/2 is symmetric).
‧A1/2 A1/2 =A.
‧(A1/2) -1 = P1 / 2 P' A1/2.
‧A1/2A-1/2= A-1/2A1/2 = I.
‧A-1/2A-1/2 =A-1.

◎Random Vectors and Matrices

⊙Mean: E(X)
⊙Variance-covariance: Σ=Cov(X)=E(X-μ)( X-μ)’
 1 12  1 p 
 1   2 p   ik
⊙Correlation Coefficient:    , ik 
21
.
       ii  kk
 
  p1  p 2  1 
  11 0  0 
 
 0  22  0 
⊙Let V 
1/ 2
, then V 1 / 2 V 1 / 2   ;   (V 1 / 2 ) 1 (V 1 / 2 ) 1 .
     
 
 0 0   pp 
⊙Linear combinations of mean vector and covariance matrix

3
 ○ E (cX 1 )  cE ( X 1 )  c1 .
○ E (aX 1  bX 2 )  cE ( X 1 )  bE ( X 2 )  a1  b2 .
○ Var (cX 1 )  E (cX 1  c1 ) 2  c 2Var ( X 1 )  c 2 11 .
○ Var (aX 1 , bX 2 )  E (aX 1  a1 )(bX 2  b2 )  abCov ( X 1 , X 2 )  ab 12 .
○ Var (aX 1  bX 2 )  E[(aX 1  bX 2 )  (a1  b 2 )]2  E[a ( X 1  1 )  b( X 2   2 )]2 .
 E[a 2 ( X 1  1 ) 2  b 2 ( X 2   2 ) 2  2ab ( X 1  1 )( X 2   2 )]
 a 2 Var ( X 1 )  b 2 Var ( X 2 )  2abCov ( X 1 , X 2 )  a 2 11  b 2 22  2ab 12 .
○Z=c’X = c1X1+c2X2+…+cpXp:
‧mean: E(Z) =E(CX)=Cμ
‧variance: Cov(Z)= Cov(CX) = CΣC’.

◎Matrix Inequalities and Maximization

⊙Discrimination analysis: To maximize the separation between groups relative to their
within-group variability.
⊙Principal components: To maximize variability of linear combinations of measurements.
⊙Cauchy-Schwarz Inequality: Let b and d are any two ( p  1 ) vectors, then (b' d ) 2  (b' b)( d ' d ) .
○Equality holds: b =cd. or b=0 or d=0.
⊙Extended Cauchy-Schwarz Inequality: (b' d ) 2  (b' Bb )( d ' B 1d ) .
( x' d ) 2
⊙Maximization lemma: max  d ' B 1d .
x0 x' Bx
x' Bx x' Bx
⊙Interpretation of eigenvalue: max  1 , attained when x=e1. min   p , attained
x0 x' x x  0 x' x

when x=ep, for 1  2  ...   p .

◎Multivariate Normal Distribution

1
⊙Univariate Normal Distribution ( N (  ,  2 ) ): f ( x)  e [( x   ) /  ]
2
/2

2 2

1 1
⊙Multivariate Normal Distribution ( N p (  , ) ): f ( X )  e  (1 / 2 )( X   )' ( X )

( 2 )
1/ 2
p/2

○Constant probability density contour: ( x   )'  1 ( x   )  c 2 (Surface of an ellipsoid
centered atμ with axes  c i ei ).
○If a random vector X having a multivariate normal distribution, the followings hold true:
‧Linear combination of the components of X are normally distributed.
‧All subsets of the components of X have a (multivariate) normal distribution.
‧Zero covariance implies that the corresponding components are independently distributed.
‧The conditional distribution of the components are (multivariate) normal.

4
 § Inferences About a Mean Vector
( x  0 ) 1 n 1 n
◎Univariate: H 0 :    0 ; H1 :   0  t 
s/n
, where x   x j ; s 
n j 1

n  1 j 1
( x j  x)2 .

( x  0 ) 2
t 2
 n( x  0 )( s ) 1 ( x  0 ) . n ( x   0 )( s ) 1 ( x   0 )  t n21 ( / 2)  Reject H0.
s/n
◎Multivariate: H 0 :   0 ; H1 :   0  T 2  n( X   0 )' S 1 ( X   0 ) , where
1 n 1 n
X  
n j 1
X j ;S   ( X j  X )( X j  X )' . T2 is called Hotelling’s T2, which is
n  1 j 1
(n  1) p
distributed as Fp ,n  p .
(n  p)
◎Hotelling’s T2 and Likelihood ratio tests
max L(  0 , )
n/2
     
⊙Likelihood ratio =    
  . 2 / n    is called Wilk’s lambda.
max L(  , )   0  
 0

 ,

1 1 n
where, L( 0 , ) 
(2 ) np / 2 
n/2
exp(   ( x j  0 )'  1 ( x j  0 ) (under hypothesis H0)
2 j 1
T 2 1
⊙   (1 
2/n
)
(n  1)
◎Generalized likelihood ratio methods
max L( )
  0
⊙Under H 0 :    ,   c.
max L( )
 
max L( )
  0
⊙When the sample size n is large,  2 ln   2 ln( ) is approximately, a  v2 v 0 .
max L( )
 
Where v-v0= (dimension of  )- (dimension of  0). v=p+p(p+1)/2; v0=0+p(p+1)/2.

◎Confidence Interval

⊙Let X1, X2,…, Xn be a random sample from an Np(μ,Σ) population withΣ positive definite,
then the 100(1-α)% confidence interval:
p ( n  1) p ( n  1)
n ( X   0 )' S 1 ( X   0 )  c 2  Fp ,n  p ( )  i c / n  i Fp ,n  p ( )
(n  p) n(n  p )
p ( n  1)
At center X, the axes of the confidence ellipsoid are  i Fp ,n  p ( ) ei
n(n  p )
○A numerical example on Page 181.

p ( n  1) s p ( n  1) s
x1  Fp ,n  p ( ) 11  1  x 1  Fp ,n  p ( ) 11
(n  p) n (n  p) n

5
 p ( n  1) s p ( n  1) s
x2  Fp ,n  p ( ) 22  1  x 2  Fp ,n  p ( ) 22
(n  p) n (n  p) n

…
p( n  1) s pp p( n  1) s pp
xp  Fp ,n  p ( )  1  x p  Fp ,n  p ( )
(n  p) n (n  p) n

⊙Simultaneous confidence interval with one-at-a-time intervals

Let Z  l1 X 1  l2 X 2  ...  l p X p  l ' X
 Z  E ( Z )  l '  ;  Z2  Var ( Z )  l ' l
sZ sZ
z  t n 1 ( / 2)   Z  z  t n 1 ( / 2)
n n
l ' Sl l ' Sl
 l ' x  t n 1 ( / 2)   Z  l ' x  tn 1 ( / 2)
n n
n(l ' ( X   )) (l ' ( X   )) 2
2
 max t 2  max = n[max ]  n( X   )' S 1 ( X   )  T 2 with
l l ' Sl l ' Sl
maximum occurring for l proportional to S 1 ( X   ) .
However, simultaneous confidence intervals for all X is (1-α)p, which is much less than
(1-α) by setting l1=[1, 0, …,0], l2=[0, 1, 0,…, 0],…, lp=[0, 0, …, 1].

⊙Simultaneous confidence interval with the Bonferroni method for multiple comparisons:
p p
P[all li true] = 1- P[at lease one li false] = 1   P[li false ] = 1   (1  P[li true ]) =
i 1 i 1

 sii  sii
1  (1   2  ...   P )  x i  t n 1 ( )   Z  x i  t n 1 ( ) , i=1,2,…,p.
2p n 2p n

p ( n  1) 
Conclusion  Choose which of Fp ,n  p ( ) and t n 1 ( ) is smaller.
n(n  p ) 2p

6
 § MANOVA
◎Paired comparisons
⊙Univariate (t-test):
○Dj= x1j – x2j, j=1,2,…,n

D  1 n 1 n
○t 
sd / n
, where, D  
n j 1
D j , s d  
n  1 j 1
( D j  D)2 .

⊙Multivariate (T-test):
(n  1) p
○ T 2  n( D   )' S d1 ( D   ) is distributed as Fp ,n  p
(n  p)
p ( n  1) sd i
○Simultaneous confidence interval: d i  Fp ,n  p ( )
(n  p) n
 sd i
○Bonferroni confidence interval: d i  tn 1 ( )
2p n
◎A repeated measures design
⊙A total of q treatments are compared with respect to a single response variable for each of the
objects.
(n  1)( q  1)
○ T 2  n(C X )' (CSC ' ) 1 (C X )  Fq 1, n  q 1 ( )
(n  q  1)
( n  1)( q  1) C ' SC
○Simultaneous confidence interval: C X  Fq 1,n  q 1 ( )
( n  q  1) n
 C ' SC
○Bonferroni confidence interval: C X  tn 1 ( )
2(q  1) n

 1   2  1  1 0  0   1 
    
1 0 1  0  2 
where C is the contrast matrix. Case 1: 
3
1
  
          
    
 1   q  1 0 0   1   3 

  2  1    1 1 0  0 0  1 
    
0 1 1  0 0   2 
Case 2: 
2 
3
  
            
    
  q   q 1   0 0 0   11   3 

◎Comparing mean vectors from two populations (both are multivariate normal)
⊙When n1 and n2 are small andΣ1=Σ2.
n1 n2

 ( x1 j  x1 )( x1 j  x1 )'   ( x2 j  x 2 )( x2 j  x 2 )'
j 1 j 1 ( n1  1) S1  ( n2  1) S 2
S pooled  
n1  n 2  2 n1  n2  2

7
 ○To test H 0 : 1   2   . E ( X 1  X 2 )  E ( X 1 )  E ( X 2 )  1   2 ;
1 1 1 1
Cov ( X 1  X 2 )  Cov ( X 1 )  Cov ( X 2 )  (  )  (  ) S pooled
n1 n2 n1 n2
1
 1 1 
○Likelihood ratio test: T  ( X 1  X 2   )'    S pooled  ( X 1  X 2   )
2

 n1 n2  
○Simultaneous confidence interval:
( n1  n2  2) p 1 1
l( X 1  X 2 )  Fp ,n1  n2  p 1 ( ) l ' (  ) S pooled l
( n1  n2  p  1) n1 n2
⊙When n1 - p and n2 – p are large andΣ1  Σ2. (Hold true even for the populations not
come from multivariate Normal)
1
1 1 
○Likelihood ratio test: T  ( X 1  X 2   )'  S1  S 2  ( X 1  X 2   )   p2 ( )
2

 n1 n2 

◎Comparing several multivariate population means (one-way MANOVA)

⊙Assumptions:
○Xl1, Xl2,…, Xlnl is a random sample of size nl from a population with mean ul, l=1, 2,…, g.
The random samples from different populations are independent.
○All populations have a common covariance matrix Σ. (i.e. Σ1=Σ2…=Σg)
○Each populations is multivariate normal. (Can be relaxed by appealing to the central limit
theorem).
⊙A summary of one-way ANOVA:
○SSobs=SSmean+SStreatment effect+SSresidual
g nl g g nl

 xlj2  (n1  n2  ...  n g ) x   nl ( x l  x ) 2   ( xlj  x l ) 2

2

l 1 j 1 l 1 l 1 j 1

○SScorrected TSS=SSbetween SS +SSwithin SS

g nl g g nl

 ( xlj  x ) 2   nl ( x l  x ) 2   ( xlj  x l ) 2
l 1 j 1 l 1 l 1 j 1

Source of variation Sum of squares (SS) Degrees of freedom

Treatment g g 1
SS tr   nl ( x l  x ) 2
l 1

Residual (error) g nl g
SS res   ( xlj  x l ) 2 n
l 1
l g
l 1 j 1

Total (corrected for the mean) g nl g

SS cor   ( xlj  x ) 2
n
l 1
l 1
l 1 j 1

SS tr
g 1
○F  as Fg 1, ( )
SS res  nl  g
n l g

8
⊙One-way MANOVA:
Source of variation Sum of squares (SS) Degrees of freedom
Treatment g g 1
B   nl ( x l  x )( xi  x )'
l 1

Residual (error) g nl g
W   ( xlj  x l )( xlj  x l )' n
l 1
l g
l 1 j 1

Total (corrected for the mean) g nl g

B  W   ( xlj  x )( xlj  x )' n
l 1
l 1
l 1 j 1

W
○Wilks’ lambda: * 
B W
○Distribution of Wilk s’ lambda:
No. of No. of Sampling distribution for multivariate normal data
variables groups
p=1 g>=2   n l  g  1   * 
 
 g  1   * 
~ Fg 1, n  g
l
 
p=2 g>=2   n l  g  1  1   * 
   ~ F2 ( g 1), 2 ( n  g 1)
 g  1   *  l
   
p>=1 g=2   n l  p  1  1   * 
 
  * 
~ Fp , n  p 1
 p l
 
p>=1 g=3   n l  p  2  1   * 
   ~ F2 p , 2 ( n  p  2 )
 p   *  l
  

⊙A summary of two-way ANOVA (Factorial ANOVA):

○SSobs=SSmean+SSfac1+SSfac2+SSint+SSres
○SScor=SSfac1+SSfac2+SSint+SSres
g b n g b g b


l 1 k 1 j 1
( xlkj  x ) 2   bn ( x l   x ) 2   gn ( x k  x ) 2   n ( x lk  x l   x k  x ) 2 +
l 1 k 1 l 1 k 1
g b n

 ( x
l 1 k 1 j 1
lkj  x lk ) 2

Source of variation Sum of squares (SS) Degrees of freedom

Factor 1 g g 1
SS fac1   bn ( x l   x ) 2
l 1

Factor 2 b b 1
SS fac 2   gn ( x k  x ) 2
k 1
Interaction g b ( g  1)( b  1)
SS int   n ( x lk  x l   x k  x ) 2
l 1 k 1

Residual (error) g b n gb( n  1)

SS res   ( xlkj  x lk ) 2
l 1 k 1 j 1

9
Total (corrected for the mean) g b n gbn  1
SS cor   ( xlkj  x ) 2
l 1 k 1 j 1

SS int
( g  1)( b  1)
○Test the interaction: F  as F( g 1)( b 1), gb ( n 1) ( )
SS res
gb( n  1)

SS fac1 SS fac 2
g 1 b 1
○Test the main factor F  as Fg 1, gb( n 1) ( ) ; F  as
SS res SS res
gb( n  1) gb( n  1)
Fb 1, gb ( n 1) ( )

⊙Two-way MANOVA (Factorial MANOVA):

○SSobs=SSmean+SSfac1+SSfac2+SSint+SSres
○SScor=SSfac1+SSfac2+SSint+SSres
Source of Sum of squares (SS) Degrees of
variation freedom
Factor 1 g g 1
SS fac1   bn ( x l   x )( x l   x )'
l 1

Factor 2 b b 1
SS fac 2   gn ( x k  x )( x k  x )'
k 1
Interaction g b ( g  1)( b  1)
SS int   n ( x lk  x l   x k  x ) ( x lk  x l   x k  x )'
l 1 k 1

Residual (error) g b n gb( n  1)

SS res   ( xlkj  x lk )( xlkj  x lk )'
l 1 k 1 j 1

Total (corrected g b n gbn  1

for the mean) SS cor   ( xlkj  x )( xlkj  x )
l 1 k 1 j 1

SS res
○Wilks’ lambda for testing interaction: * 
SS int  SS res
 p  1  ( g  1)( b  1) 
  gb( n  1)   ln  * as  (2g 1)( b 1) p ( )
 2 
SS res
○Wilks’ lambda for testing factor 1: * 
SS fac1  SS res
 p  1  ( g  1) 
  gb( n  1)   ln  * as  (2g 1) p ( )
 2 
SS res
○Wilks’ lambda for testing factor 2: * 
SS fac 2  SS res
 p  1  (b  1) 
  gb( n  1)   ln  * as  (2b1) p ( )
 2 

10
 § Multivariate Linear Regression Models

◎Univariate Linear Regression

⊙ Y   0  1 x1   2 x 2  ...   r x r  
With n samples, it can be expressed as:
Y1    0  1 x11   2 x21     r xr1   1 
Y      x   x     x   
 2   0 1 12 2 22 r r2 2

   
   
 n  0
Y    x
1 1n   x
2 2n     x
r rn   n

In matrix form:
Y1  1 x11  x r1    0    1 
Y  1 x  x r 2   1   2 
 2   12     
         
       
Yn  1 x1n  x r 3    r   n 
Y X    
[n  1][ n  ( r  1)][( r  1)  1][ n  1]
where, the error term are assumed to have the properties:
E ( )  0 ; Cov ( )  E ( ' )   2 I (constant).
⊙Multiple regression vs. ANOVA (using three dummy variables for three level factor)
⊙Least squares estimation
○To minimize the error sum of squares (ESS):
Min ESS   '  (Y  X )' (Y  X )  ˆ  ( X ' X ) 1 X ' Y distributed as
N r 1 (  ,  2 ( X ' X )  1 )
n n n
○Decomposition:  ( y j  y ) 2   ( yˆ j  y )   ˆ 2j
j 1 j 1 j 1

total sum regression residual (error)

of squares = sum of + sum of squares
about mean squares
TSS = RSS + ESS
○The confidence ellipsoid for  is:
(   ˆ )' X ' X (   ˆ )  ( r  1) s 2 F ( ) r 1, n  r 1

○Simultaneously 100(1-α)% confidence interval:

ˆ  Var (  ) ( r  1) F
i i ( ) r 1, n  r 1

○One-at-a-time 100(1-α)% confidence interval:


ˆi  t n  r 1 ( ) Var (  i )
2
○F-test for the significance of more than two coefficients
H 0 :  q 1   q  2  ... r  0
SS res ( X 1 )  SS res ( X ) /( r  q)
 Fr  q ,n  r 1 ( )
SS res ( X ) /( n  r  1)
X 1  [1, x1 , x2 ,..., x q ] ; X 2  [ x q 1 , x q  2 ,..., x r ]
○R-square = SSR/TSS = (1-ESS)/TSS

11
 ○Adjusted R-square = 1-(1-R2)(n-1)/(n-r-1)

○Estimation confidence interval: X 0 ' ˆ  t n  r 1 ( ) X 0 ' ( X ' X ) 1 X 0 
2

○Prediction confidence interval: X 0 ' ˆ  t n  r 1 ( ) (1  X 0 ' ( X ' X ) 1 X 0 )
2
ˆ
Var (Y0  X 0 '  )   (1  X 0 ' ( X ' X ) X 0 )
2 1

⊙Model checking
○Plot residuals
‧against the predicted values y.
‧against the predictor variable x.
‧against time (sequence of observations).
‧Q-Q plot (normal distribution check).
○Multicollinearity  (X’X)-1 is nearly zero and numerically unstable.  VIF and TOL.
n n
○Serial correlation  Durbin-Watson test.  ( j   j 1 ) 2 /   2j
j 2 j 1

○Homogeneity  Goldfeld-Quandt test.

◎Multivariate Linear Regression

⊙Modeling the relationship between m responses, y1, y2,..., ym and a single set of predictor
variables, x1, x2, …, xr :
y1   01  11 x1   21 x 2  ...   r1 x r   1
y 2   02  12 x1   22 x 2  ...   r 2 x r   2
.
.
.
y m   0 m  1m x1   2 m x 2  ...   rm x r   m

Y  X   with E ( i )  0 ; Cov ( i ,  k )   ik2 I

⊙Multivariate linear regression has no new estimation problems. The parameters in each
repression equation can be estimated and diagnosed individually for each response variable.
⊙The model requires the same predictor variables be used for all responses.

Source of Sum of squares (SS) Degrees of freedom E (SS & CP)

variation
Regression R k 1
k   ' X ' ( I n  11' ) X
n
Error E n-p-1 (n  p  1)
Corrected Total T n-1

⊙Model overall fit

p
E
Wilks’ ratio:     (1  i ) 1
RE i 1

12
 R 1 1 p i
Pillai’s trace: ; tr[ R( R  E ) ]  
1
(The same way of univariate
RE p p i 1 1  i
R-square)
1 1 p
Hotelling-Lawley trace: tr ( E 1R )   i
p p i 1
Roy’s maximum root: max  max( 1 , 2 ,...,  p )  1
All of above statistics can be approximated by F distribution.

⊙Likelihood ratio tests for regression parameters

  ( 1) 
H 0 :  (2)  0    
 (2) 
max L(  (1) , )
  
n/2
 ( 1) ,
   
max L(  , )  1 
 ,

   nˆ

 2 ln    n ln  
   n ln nˆ  n (ˆ  ˆ )
 1  1

1   
For n large, the modified statistic:  [n  r  1  ( m  r  q  1)] ln   ~  m2 ( r  q ) .

2  1 
⊙Predicted values
Yˆ  Xˆ  X ( X ' X ) 1 X ' Y
ˆ  Y  Yˆ  Xˆ  X ( X ' X ) 1 X ' Y
X ' ˆ  X ' [ I  X ( X ' X ) 1 X ' ]Y  0
Yˆ ' ˆ  ˆ ' X ' [ I  X ( X ' X ) 1 X ' ]Y  0
Y ' Y  (Yˆ  ˆ )(Yˆ  ˆ )  Yˆ ' Yˆ  ˆ' ˆ  0  0
total sum predicted sum of residual (error)
of squares = squares and cross- + sum of squares and
and cross-products product cross-product

⊙Example 7.8 (on page 302)

⊙The 100(1-α)% simultaneous confidence intervals for E (Yi )  x0  ( i )
 m( n  r  1)   n 
x0 ' ˆ( i )    Fm ,n  r  m ( ) x0 ( X ' X ) x0 
1
ˆ ii  , i=1, 2,.., m.
 nrm   n  r 1 
⊙The 100(1-α)% simultaneous prediction intervals for Y0i
 m( n  r  1)   n 
x0 ' ˆ( i )    Fm ,n  r  m ( ) (1  x0 ( X ' X ) x0 )
1
ˆ ii  , i=1, 2,.., m.
 nrm   n  r 1 

◎Simultaneous-Equation Models
⊙Structural equations: BYt  X t  ut , t=1, 2, …, T
⊙With G endogenous variables and K exogenous variables.

13
⊙Reduced form: Yt  X t  vt , t=1, 2, …, T (Only for B is a nonsigular matrix)
⊙Estimation methods:
○Single equation method: estimate each equation with related restrictions separately by least
squares method. Y  Y1   X 1  u （Y1：other included endogenous variables）
○ System method: estimate all equations with all restrictions together by maximum
likelihood method.
○Two-stage least squares (2SLS)
○Limited-information maximum-likelihood method
○Three-stage least squares (3SLS)
○Full-information maximum-likelihood method

14
 § Discrimination Analysis

◎Separation and classification for two populations: Fisher’s method

⊙Populations:  1 ,  2 with measured variables: X (x1, x2,…, xp)
⊙The core logic of Fisher’s method: transform the multivariate observations X to univariate
observations y such that the y’s derived from populations  1 ,  2 were separated as much as
possible.
○ 1  E ( X  1 ) : Expected value of a multivariate observation from  1
○  2  E ( X  2 ) : Expected value of a multivariate observation from  2
○   E ( X  1 )( X  2 )'
○A linear combination Y  l ' X
○ 1Y  E (Y  1 )  E (l ' X  1 )  l ' 1 or  2Y  E (Y  2 )  E (l ' X  2 )  l '  2
○  Y2  Var (l ' X )  l ' l
○The best linear combination is derived from the ratio (Fisher’s method):
○(Squared distance between means of Y)/(variance of Y) =
( 1Y   2Y ) 2 (l ' 1  l '  2 ) 2 l ' ( 1   2 )( 1   2 )' l (l '  ) 2
   (   ( 1   2 ) )
 Y2 l ' l l ' l l ' l
(l '  ) 2
 Max   '  1 by setting l  c 1 ( 1   2 ) .
l ' l
 By setting c=1, Y  l ' X  ( 1   2 )'  1 X
○The midpoint between two population univariate means:
1 1 1
m  ( 1Y   2Y )  (l ' 1  l '  2 )  ( 1   2 )'  1 ( 1   2 )
2 2 2
For a new observation x0: (Figure 11.1 on page 476)
 Allocate x0 to π1 if E (Y0  1 )  m  0 or y0  ( 1   2 )'  1 x0  m
 Allocate x0 to π2 if E (Y0  2 )  m  0 or y0  ( 1   2 )'  1 x0  m
 n1  1   n2  1 
○ S pooled    S1   S2
 ( n1  1)  ( n2  1)   ( n1  1)  ( n2  1) 
○ANOVA test for the means. (F-test)
○Significant separation does not necessarily imply good classification.
○A good classification procedure should result in few misclassifications.
○The scaled or normalized technique is use to ease the interpretation of the vector l. The
lˆ
commonly used normalization: lˆ *  . l* has unit length.
lˆ' lˆ
○Normalization is recommended only if X variables have been standardized.
◎Optimal classification rules for two populations
⊙Consider the prior probability of occurrence.
○P(correctly classified asπ1)= P ( X  R1  1 ) P ( 1 )  P (11) p1
○P(misclassified asπ1)= P ( X  R1  2 ) P ( 2 )  P (1 2) p2

15
 ○P(correctly classified asπ2)= P ( X  R2  2 ) P ( 2 )  P ( 2 2) p 2
○P(misclassified asπ2)= P ( X  R2  1 ) P ( 1 )  P ( 2 1) p1
⊙Consider misclassification costs are significantly different.
Classify as:
π1 π2
π1 0 C ( 2 1)
True population:
π2 C (1 2) 0
⊙Expected cost of misclassification (ECM)= c( 2 1) P ( 2 1) p1  c(1 2) P (1 2) p 2
f1 ( x )  c(1 2)   p2  f1 ( x )  c(1 2)   p2 
⊙Region R1 & R2  R1 :     ; R2 :  
f 2 ( x )  c( 2 1)   p1  f 2 ( x )  c( 2 1)   p1 
where, fi(x) is the p.d.f of the population i.

◎Classification with two multivariate normal population

For 1   2
1 1  c(1 2)   p2 
⊙ R1 : exp[  ( x  1 )'  1 ( x  1 )   ( x   2 )'  1 ( x   2 )]    
2 2  c( 2 1)   p1 
1 1  c(1 2)   p2 
⊙ R2 : exp[  ( x  1 )'  1 ( x  1 )   ( x   2 )'  1 ( x   2 )]    
2 2  c( 2 1)   p1 
For 1   2 (Quadratic classification rule)
1  c(1 2)   p2 
⊙ R1 :  x0 ' (11   21 ) x0  ( 1 ' 11   2 '  21 ) x0  k  ln   
2  c(2 1)   p1 
1  c(1 2)   p2 
⊙ R2 :  x0 ' (11   21 ) x0  ( 1 ' 11   2 '  21 ) x0  k  ln   
2  c(21)   p1 
1   1
where, k  ln  1   ( 1 ' 11 1   2 '  21  2 )
2  2  2
⊙Performance evaluation: Error rate & Hit ratio.
○Training sample and validation sample (error rate determined by validation sample).
○Handout procedure (taking one sample out one-at-a-time)(Leave-one-out cross-validation)
○K-fold cross validation

◎Classification among several population

⊙Fisher’s sample discriminants: Let 1 , 2 ,..., s denote the s  min( g  1, p ) nonzero
eigenvalues of W-1B and e1, e2, …, es be the corresponding eigenvectors (scaled so that
e’Spoolede=1). Then the vector of coefficients l that maximizes the ratio:
g
lˆ'[ ( x i  x )( x i  x )' ]lˆ
lˆ' Blˆ
 i 1
is given by l1=e1.
ˆl 'Wlˆ g ni
ˆl '[
 ( xij  xi )( xij  xi )' ]l
i 1 j 1
ˆ

where, B is sample between variation. W is sample within variation.

16
 ○The linear combination l '1 x is called the sample first discriminant.
○The linear combination l '2 x is called the sample second discriminant.
○The linear combination l 'k x is called the sample kth discriminant.
⊙The minimum expected cost of misclassification method
○ ECM  p1ECM (1)  p2 ECM (2)  ...  pg ECM ( g )
g g g 1
 p1 ( P (k 1)c(k 1))  p2 ( P (k 2)c(k 2))  ...  pg ( P (k g )c(k g ))
k 2 k 1 k 1
k 2
g g
  pi ( P ( k i )c ( k i ))
i 1 k 1
k i
g
○Allocate x to  k for which  p f ( x )c ( k i )
k 1
i i is smallest.
k i

○Classification with Normal distribution with equal misclassification cost:

 p 1 1
○ ln pk f k ( x)  ln pk    ln 2  ln  k  ( x   k )'  k ( x   k )  max ln pi f i ( x)
2 2 2 i

1 1
○The quadratic discrimination score: d kQ ( x)  ln  k  ( x   k )'  k ( x   k )  ln pi
2 2
1
○The linear discrimination score: d kQ ( x )   k'  1 x   k '  k  k  ln pi (if  i   )
2

○Allocate x to  k if the score of d kQ (x) = largest of d1Q ( x ), d 2Q ( x ),..., d gQ ( x ) .

17
 § Logistic Regression
◎ Classification with two population & without assumptions of normality of exploratory
variables.
⊙Linear regression models of the form: E(y |x1, . . . , xk) = β0 + β1x1 + ···+ βkxk
⊙Let Pr(y = 1|x1, x2,. . . , xk) as the outcome, which is equal to E(y|x1, . . . , xk) when the
response (y) is coded as 0 or 1.
⊙To simplify notation, let the outcome of our model be π(X) = Pr(y = 1|x1, . . . , xk), where x
represents x1, x2,. . . , xk.
⊙π(X) represents the probability of the outcome of interest and lies in the interval [0, 1].
⊙π(X)/(1 − π(X)) belongs to the interval (0,∞).
⊙log{π(X)/(1 − π(X))} belongs to the interval (−∞,∞); this is the same range of values to
which the expression β0 + β1x1 + ···+ βkxk belongs.
⊙The basis for logistic regression is the equation (statistical model) log{π(X)/1-π(X)} =β0
+ β1X1 + · · · + βkXk.
⊙π(X)/1−π(X) is the odds of the outcome of interest for an individual with covariates X.
⊙log {π(X)/1−π(X)} is called the log odds or the logit.

18
 § Principal Components

◎Definition: Principal components aims to explain the variance-covariance structure through a

few linear combinations of the original variables.  Dimension reduction.
◎Let the random variables X = [x1, x2,…, xp] have the variance-covariance matrix  .
⊙Consider the linear combinations:
y1  l1 ' X  l11 x1  l21 x2  ...  l p1 x p
y2  l2 ' X  l12 x1  l22 x2  ...  l p 2 x p
…
y p  l p ' X  l1 p x1  l2 p x2  ...  l pp x p
so that, Var ( yi )  li ' li , i  1,2,..., p ; Cov ( yi , yk )  li ' lk  0, i, k  1,2,..., p
⊙The principal components are those uncorrelated linear combinations y1, y2, …, yp whose
variances are as large as possible.
○First principal component = linear combination l1 ' X that maximizes Var ( l1 ' X ) subject
to l1 ' l1  1 .
○Second principal component = linear combination l2 ' X that maximizes Var ( l2 ' X )
subject to l2 ' l2  1 and Cov (l1 ' X , l2 ' X )  0 .
○ith principal component = linear combination li ' X that maximizes Var ( li ' X ) subject to
li ' li  1 and Cov (lk ' X , li ' X )  0 for k < i.
⊙Let 1  2  ...   p  0 denote the nonzero eigenvalues of  with corresponding
eigenvectors of e1, e2, …, ep. The ith principal component is given by:
yi  ei ' X  e1i x1  e2i x2  ...  e pi x p , i=1, 2, …, p with Var ( yi )  ei ' ei , i  1,2,..., p and
Cov ( yi , yk )  ei ' ek  0, i  k .
l ' l
○Proof: Since max  1 (attained when l=e1). Since e1’e1=1 (normalized),
l 0
l'l
l ' l e ' e l ' l
max  1  1 1  e1 ' e1  Var ( y1 ) . max  k 1 , k =1, 2, …, p-1. For the
l 0 l'l e1 ' e1 l  e1 , e 2 ,...,e k l ' l

e ' e
choice of l=ek+1 and e’k+1ek=0, k=1, 2, …, p-1. k 1 k 1  k 1  Var ( yk 1 ) .
ek 1 ' ek 1
○Total population variance =
p p
 11   22  ...   pp   Var ( xi )  1  2  ...   p   Var ( yi ) .
i 1 i 1

○The proportion of total variance due to (explained by) the kth principal component =
k
.
1  2  ...   p
eki i
⊙The correlation coefficients between the components yi and the variables xk:  y i , x k  ,
 kk
i, k =1, 2, …, p.

◎If the random variables X = [x1, x2,…, xp] is distributed as N p (  , ) , then  is centered

19
 ellipsoids: ( X   )'  1 ( X   )  c 2 of constant density have axes  c i ei , i=1, 2, …,p. Let
1 1 1
 =0, c 2  X '  1 X  (e1 ' X ) 2  (e2 ' X ) 2  ...  (e p ' X ) 2 
1 2 p
1 1 1
c2  y1  y2  ... 
2 2 2
yp (Figure 8.1 on page 346)
1 2 p

◎Principal components obtained from standardized variables

( x  i )
⊙The random variables Z = [z1, z2,…, zp], where zi  i . E(zi)=0 and Cov ( zi , z j )   .
 ii
⊙In matrix form: Z  (V 1 / 2 ) 1 ( X   ) , where V1/2 the diagonal standard deviation matrix. (on
page 57).
⊙E(Z)=0 and Cov ( Z )  (V 1 / 2 ) 1 (V 1 / 2 ) 1   .
⊙The ith principal component of the standardized variables Z with Cov (Z )   is given by:
p p
yi  ei ' Z  ei ' (V 1 / 2 ) 1 ( X   ) , i=1, 2, …, p. Var (yi )  Var ( zi )  p .
i 1 i 1

○The proportion of (standardized) population variance due to the kth principal component =
k
. 1  2  ...   p  0 are the eigenvalues of  .
p

20
 § Factor Analysis

◎Definition: Factor analysis can be considered as an extension of principal components by

providing more elaborate analysis on variance-covariance structure.

◎The orthogonal factor model: x    LF  

Z1= x1  1  l11F1  l12 F2  ...  l1m Fm   1
Z2= x2   2  l21F1  l22 F2  ...  l2 m Fm   2
…
Zp= x p   p  l p1F1  l p 2 F2  ...  l pm Fm   p
Fi: common factors.  i : specific errors. lij is called the loading of the ith variable on the jth
factor. E(F)=0 and Cov(F)=E(FF’)=I. E(ε)=0 and
 1 0  0 
0   0 
Cov ( )  E ( ' )      (diagonal matrix) Cov ( , F )  E (F ' )  0 .
2

    
 
 0 0   p 

⊙ ( X   )( X   )'  ( LF   )( LF   )'  LF ( LF )' ( LF )' LF ' '

So that
  E ( X   )( X   )'  LE ( FF ' ) L' E (F ' ) L' LE ( F ' )  E ( ' )  LL '

⊙ ( X   ) F '  ( LF   ) F '  LFF 'F ' , so Cov ( X , F )  E ( X   ) F '  LE ( FF ' )  E (F ' )  L .
⊙Covariance structure for the Orthogonal Factor Model:
○ Cov ( X )  LL '
Var ( xi )  li1  li 2  ...  lim   i
2 2 2

Var(xi)= communality + specific variance

Cov ( xi , x j )  li1l j1  li 2l j 2  ...  liml jm
○ Cov ( X , F )  L
Cov ( xi , F j )  lij

◎Methods of Estimation
⊙Principal component method
⊙Maximal likelihood method

◎Factor Rotation
⊙L*=LT, where TT’=T’T=I. The estimated covariance (or correlation) matrix remains
unchanged. LL '  L'TT ' L    L * L*' .
 cos  sin  
Taking m=2 at a time, L *  L T where T    (clockwise rotation);
( p 2) ( p  2 ) ( 2 2 )
 sin  cos  
cos   sin  
T   (counterclockwise rotation).
 sin  cos  

21
 ⊙Rotated loadings must be inspected to find a meaningful interpretation of the original data.
⊙Varimax rotation (proposed by Kaiser,1958) selects the orthogonal transformation T that
  p *2  
2

   lij  
1 m  p *4  i 1  
makes: V    lij   as large as possible.
p j 1 i 1 p
 
 

◎Factor Scores
⊙The estimated value of common factors are called factor scores.
⊙Usually, the estimated rotated loadings are used to compute factor score.
⊙The weighted least square method:
f  ( L' 1 L ) 1 L 1 ( x   )
If factor loadings are estimated by the principal component method: f  ( L' L) 1 L( x   )
⊙The regression method: f  L' S 1 ( x   ) .

22