• Consider
two
a:ributes
X
and
Y
• Each
is
sampled
three
?mes
! x $ ! $ ! y $ ! $
# 1 & # 1 & # 1 & # −1 &
X = # x2 & = # 0 &; Y = # y2 &=# 1 &
# & # −1 & # & # 0 &
#" x3 &% " % #" y3 &% " %
• Combine
these
two
into
a
matrix
S
! x y1 $ ! 1 −1 $
# 1 & # &
S = # x2 y2 & = # 0 1 &
# &
#" x3 y3 &% #" −1 0 &%
• Compute
the
(un-‐normalized)
covariance
matrix,
C
" 1 −1 % "σ σ %
" 1 0 0 % $ ' " %
2 −1 $ xx xy
T
C = S S =$ ' $ 0 1 '=$ '= '
# −1 1 −1 & $ −1 0 ' # −1 2 & $# σ yx σ yy '
&
# &
• Compute
the
eigenvalues
and
eigenvectors
of
C
by
solving
Cu = λu or (C − λ I)u = 0
• If
all
the
a:ribute
vectors
are
independent,
then
we
have
M=2
eigenvalues
and
M
=
2
eigenvectors
• Because
the
Correla?on
matrix
is
real
and
symmetric
all
the
eigenvalues
are
real
and
posi?ve
• Plugging
in
the
numbers,
we
get
" 2−λ % " u % "
−1 $ 1 ' 0 %
$ ' =$ '
# −1 2 − λ &$# u2 '& # 0 &
• To
solve
for
the
eigenvalues,
we
use
the
determinant
of
the
matrix
to
get
a
quadra?c
equa?on
2 2
(2 − λ ) −1 = λ − 4λ + 3 = 0 ⇒ λ1 = 3, λ2 =1
•
• Note
that
the
lambda’s
are
listed
in
descending
order
of
magnitude
• To
solve
for
the
eigenvectors,
we
simply
subs?tute
the
two
eigenvalues
into
the
matrix
equa?on.
• It
is
also
general
prac?ce
to
find
the
simplest
eigenvector
in
each
case
by
normalizing
it
so
that
the
sum
of
the
squares
of
its
components
equals
1.
Thus,
we
get:
Eigenvectors
• For
λ
=
1
" 1 −1 %" u1 % "
0 %
$ '$ '=$ '
# −1 1 &$# u2 ' # 0 &
&
which
gives
u1
=
u2
• For
λ
=
3
" −1 −1 %"$ u1 % " 0 %
'=$
$ ' '
$ '
# −1 −1 &# u2 & # 0 &
which
gives
u1
=
-‐u2
Normalized
Eigenvectors
• It
is
customary
to
normalize
the
eigenvectors
• The
normalized
eigenvectors
are
! u $ ! 1 $ ! v $ ! 1 $
# 1
&= 1 # 1
&= 1
# &; # &
# u2 & 2 " −1 % # v2 & 2" 1 %
" % " %
• Note
that
u
is
the
eigenvector
associated
with
larger
eigenvalue
Principal
Components
• Now
we
are
in
a
posi?on
to
compute
the
principal
components
of
S.
• The
principal
components
are
created
by
mul?plying
the
components
of
each
eigenvector
by
the
a:ribute
vectors
and
summing
the
result.
• That
is,
for
the
two
principal
components,
P1
and
P2,
we
can
write
P
1
=
u1
X
+
u2
Y
,
and
P
2
=
v1X
+
v2Y.
In
Matrix
Nota?on
• In
matrix
form,
the
principal
component
matrix
is
the
product
of
the
a:ribute
matrix
S
and
the
eigenvector
matrix
U:
! x y1 $! ! 1 $
# 1 & u1 $ −1
v1 # &! 1 1 $
P = SU = # x2 y2 &# &= 1 # 0 1 &# &
# &#" u2 v2 &% 2 # −1 0 &" −1 1 %
#" x3 y3 &% " %
• Therefore
" 2 % " 0 %
1 $ ' 1 $ '
P1 = $ −1 '; P2 = $ 1 '
2 $ −1 ' 2 $ −1 '
# & # &
• Note
also
that
the
principal
component
matrix
has
the
property
that
when
it
is
mul?plied
by
its
transpose
we
recover
the
eigenvalues
in
diagonal
matrix
form:
" 2 0 %
T 1 " 2 −1 −1 %$ ' " 3 0 %
P P= $ '$ −1 1 ' = $ '
2 # 0 1 1 &$ ' # 0 1 &
# −1 1 &
• Because
the
inverse
and
transpose
of
the
eigenvector
matrix
are
iden?cal,
we
can
write
PU T = SUU T = SUU −1 = S ⇒
# 2 0 &# # 1 −1 &
1% ( 1 −1 & % (
S = % −1 1 (% (=% 0 1 (
2% ($ 1 1 ' % (
$ −1 −1 ' $ −1 0 '
• These
columns
are
the
values
of
X
&
Y
recovered!