Principal Component Analysis
Prabhat Kumar Mishra
Example: Teleoperation
• You wish to teleoperate a robotic
hand
• You must nd a way to map human
hand’s joints onto the robotic hand’s
joints
• Several of the human hand’s joint will
map to the motion of a single robotic
joint
• Joints move in coordination. This
coordination is referred to as synergy.
• Synergies across joints can be
constructed by projecting joint angles
into lower dimensional subspace
using Principal Component Analysis
(PCA)
fi
Correspondence problem in robotics
• How to transfer motion from human to robot when
both di er in the kinematic and dynamic of their bodies
h
• Human’s joint measurement q , human’s end-point h r
position x h min
r
∥x − x ∥
q
r
• Robot’s joint measurement q , human’s end-point Subject to x h
= h h
f (q )
r
position x r r r
x = f (q )
• The optimization problem is di cult when two systems
are dramatically di erent
• Example: The human hand has typically 22-28 DoF,
most robotic hands have around 5 DoF (one for each
nger)
fi
ff
ff
ffi
Example: Teleoperation
h D
• q ∈ ℝ represents D human joints
• Robot has p joints
h p
• We construct y = Aq ∈ ℝ , where A is the projection
matrix found after performing PCA.
• Example: y1 = 1 and all other entries are zero. This will
activate the rst combination of joints corresponding to
the rst row of A.
fi
fi
Principle Component Analysis
• Also known as Karhunen-Love transform
• Orthogonal projection of data onto the lower
dimensional space
D
• xi ∈ ℝ is a data sample, we want it to be
represented in M < D dimensional space.
• Dimensionality reduction
Principle Component Analysis
D
Case M = 1: a line spanned by the vector v ∈ ℝ
⊤
Projection of xi onto v is v xi
Suppose we have N data samples, so empirical
N
1
∑
mean will be x̄ = xi
N i=1
Principle Component Analysis
•The empirical variance of the projected data
N
⊤ 1 ⊤ ⊤ ⊤ ⊤ ⊤
∑
var(v x) = (v xi − v x̄)(v xi − v x̄)
N i=1
N
( N i=1 )
⊤ 1 ⊤ ⊤
∑
=v (xi − x̄)(xi − x̄) v =: v Sv
•
Principle Component Analysis
⊤
•Let us maximize var(v x) under the equality constraint
⊤
v v=1
•Construct the Lagrangian with Lagrangian coe cient λ:
⊤ ⊤
•ℒ(v, λ) = v Sv + λ(1 − v v) and di erentiate it with
respect to v, λ, make the derivative zero to compute the
values of v, λ
ff
ffi
Principle component analysis
Maximize the variance of the
projected data
⊤
max v Sv
v
⊤
s.t. v v=1
What solution of the above optimization means
• Solution that we got is Sv = λv, where S is the
variance of the data
• The above identity tells that (v, λ) is an eigenvalue,
eigenvector pair of S
• If I project data in the direction of one eigenvector,
the variance of the projected data will be the
corresponding eigenvalue of S
• To get the maximum variance, I will choose the
largest eigenvalue
What solution of the above optimization means
• Variance is an indicator of the spread of data
• We can derive PCA as a dimensionality reduction
technique that maximises the variance in the low-
dimensional representation of the data to retain as
much information as possible
Principle component analysis
Maximize the variance of the Generalize this idea for
projected data M<D
We can get M vectors
⊤
max v Sv (corresponding to M
v
⊤
largest eigenvalues of S)
s.t. v v=1 and data will be projected
in a space spanned by
these vectors
Example
MNIST data set
• 60000 examples of handwritten digits
• Grayscale image of size 28 × 28
784
•x ∈ ℝ
Example
MNIST data set: rst two principle components
fi
Example
Eigenvalues of MNIST 8
Example
MNIST digits: reconstruction
M-dimensional subspace with maximal variance
• The maximum variance that PCA can capture by M
M
∑
principle components is VM = λi
i=1
Example
• Suppose you have collected data from 200 people about their height and
weight (you have two features about each person)
• construct a 200 × 2 matrix X to store your data
• Compute X0 = X − mean(X)
1 ⊤ 2×2
• Compute cov(X) = X0 X0 ∈ ℝ
199
• Suppose cov(X) has two di erent eigenvalues
• The eigenvector corresponding to the largest eigenvalue is principle
⊤
component (for example [0.8 0.6] )
• You can reduce dimensionality of data by projecting it in the direction of the
principle component and get new feature (0.8 height + 0.6 weight )
ff
Key steps of PCA
• Mean subtraction
• Standardization
• Eigendcompostion of the
covariance matrix
• Projection
• Undo standardization
Key steps of PCA
• Mean subtraction
• Standardization
• Eigendcompostion of the
covariance matrix
• Projection
• Undo standardization
Key steps of PCA
• Mean subtraction
• Standardization
• Eigendcompostion of the
covariance matrix
• Projection
• Undo standardization
Key steps of PCA
• Mean subtraction
• Standardization
• Eigendcompostion of the
covariance matrix
• Projection
• Undo standardization
Key steps of PCA
• Mean subtraction
• Standardization
• Eigendcompostion of the
covariance matrix
• Projection
• Undo standardization
Key steps of PCA
• Mean subtraction
• Standardization
• Eigendcompostion of the
covariance matrix
• Projection
• Undo standardization
More intuition by a toy example
Experiment by a new student
• Measure ball’s
position in 3D
• Using three cameras
(120 Hz) record
movement of the
system
• Which dimension is
important?