JMoon KAIST EE
WEEK 4
Data Preprocessing
Week of 3/17
Lecture on Tuesday 3/18 and practice session 3/20
EE214: Machine Learning Basics and Practices
SP2025
JMoon KAIST EE
Contents
• Scaling and shifting of features
• Principal component analysis (PCA) for
dimension reduction
• Non-negative matrix factorization (NMF)
for sparse data decomposition
• t-SNE dimension reduction for data
visualization
• One-hot-encoding
2
JMoon KAIST EE
Scaling and Shifting the Features
• Many algorithms are sensitive to the scaling of the data adjust the
features via scaling and shifting (so all have zero mean and unit
variance in the case of the standard scaler, for example)
• Need to apply the same adjustment to both training and test datasets
data with two features (why rescale?)
loss vs weights along feature dimensions
Unbalanced feature scales lead
to unbalanced weight scales;
GD exhibits different speeds Fig: J. JORDAN
along weight directions with
different scales, causing an
oscillatory movement and thus
a slower convergence.
Different ways to rescale and preprocess a dataset [FIG: Muller and Guido]
3
JMoon KAIST EE
Scaling and Shifting the Features
Why scaling?
• Every feature is to be treated equally importantly (to prevent one
feature from dominating)
• Convergence is faster
Min-Max Scaler:
scales and shifts each feature separately such that it is in a given range
like [0,1] or [-1,1]
Standard Scaler:
forces each feature to have zero-mean and unit variance; not the
best scaler to use, if the data is not normally distributed (Gaussian)
4
JMoon KAIST EE
Scaling and Shifting the Features
Some people mention normalization and standardization to mean the same thing, but some others
are more strict about the definitions:
Normalization: bound the data values, typically, between [0,1] or [-1,1]
Standardization: force the data to have zero mean and unit variance
Baijayanta Roy
5
JMoon KAIST EE
Principal Component Analysis (PCA):
• PCA is used to reduce dimensionality
• high dimensionality = large number of features
• high dimensionality overfitting (for a given model complexity)
Given a fixed number of training samples,
classifier performance improves with
dimension (# of features) first and then
deteriorates after a certain point.
• Curse of dimensionality:
As dimensionality increases, amounts of computation increase exponentially.
• PCA reduces the number of the
input nodes of the model (or
minimizes the dataset to train the
model)
• PC is a linear combination of
features
6
JMoon KAIST EE
Principal Component Analysis (PCA):
covariance matrix The covariance matrix is given by
the spread shape of the data.
Diagonal terms of the matrix
measure variances of the individual
features separately, while the off
diagonal terms represent
correlations across different
features.
Given N data samples 𝒙𝒙1 , 𝒙𝒙2 , …, 𝒙𝒙𝑁𝑁
(with K=2 dimensions for each
sample here), the covariance
matrix is (means are all zero here):
1
𝚺𝚺 = 𝑁𝑁 ∑𝑁𝑁
𝑛𝑛=1(𝒙𝒙𝑛𝑛 −𝒎𝒎)(𝒙𝒙𝑛𝑛 −𝒎𝒎)
𝑇𝑇
1
= 𝑁𝑁 ∑𝑁𝑁 𝑇𝑇
𝑛𝑛=1 𝒙𝒙𝑛𝑛 𝒙𝒙𝑛𝑛
Σ𝑖𝑖𝑖𝑖 : ij-th element of covariance 𝚺𝚺
= averaged product of data component
i and data component j (averaged
across data samples)
= correlation between data
components i and data components j
7
JMoon KAIST EE
Principal Component Analysis (PCA):
Eigenvectors and Eigenvalues: Math behind
Eigenvalues and eigenvectors are key summarizing information about a matrix
Given a 𝐾𝐾 × 𝐾𝐾 covariance matrix 𝑨𝑨, there exists a vector 𝒗𝒗 and a scalar
value 𝜆𝜆 such that
𝑨𝑨𝑨𝑨 = 𝜆𝜆𝒗𝒗,
which states 𝑨𝑨𝑨𝑨, a linear transformation of 𝑨𝑨, is a vector along 𝒗𝒗 with
strength 𝜆𝜆. This says that 𝑨𝑨 has a component along 𝒗𝒗 and its strength is 𝜆𝜆.
This vector 𝒗𝒗 and its corresponding scalar 𝜆𝜆 are called the eigenvector and
eigenvalue of 𝑨𝑨, respectively. There are multiple such pairs in general:
𝑨𝑨𝒗𝒗𝑖𝑖 = 𝜆𝜆𝑖𝑖 𝒗𝒗𝑖𝑖 𝑖𝑖 = 1, 2, 3, … , 𝐾𝐾
and different eigenvectors 𝒗𝒗𝑖𝑖 and 𝒗𝒗𝑗𝑗 are orthogonal.
8
JMoon KAIST EE
Principal Component Analysis (PCA):
Graphical understanding of Eigenvectors and Eigenvalues
𝒗𝒗1 , 𝒗𝒗2 are eigenvectors here
T
10 while 𝝀𝝀1 , 𝝀𝝀2 are the corresponding
1 0T eigenvalues.
01 T
01 T
Rahul Bohare
３ ２ １
＝５ １
２ ３ １ １
３ ２ −１ −１ 9
＝１
２ ３ １ １
JMoon KAIST EE
Principal Component Analysis (PCA):
Computing Eigenvectors and Eigenvalues: Example
Finding eigenvectors and eigenvalues is easy using library modules and functions,
but a simple exercise would be instructive.
𝑨𝑨𝑨𝑨 = 𝜆𝜆𝒗𝒗
𝑣𝑣1
𝑨𝑨 − 𝜆𝜆𝑰𝑰 𝒗𝒗 = 𝟎𝟎 𝑣𝑣2
nonzero solution 𝒗𝒗 𝑣𝑣1 𝑣𝑣2 𝑣𝑣
exists iff 𝑣𝑣1 𝑣𝑣2
𝑑𝑑𝑑𝑑𝑑𝑑(𝑨𝑨 − 𝜆𝜆𝑰𝑰) = 0
𝑣𝑣1
𝑣𝑣2
Each eigenvector is often normalized
further to have a magnitude of 1.
𝑣𝑣
𝑣𝑣1
𝑣𝑣2
𝑣𝑣1
𝑣𝑣2
𝑣𝑣1 𝑣𝑣2 𝑣𝑣
𝑣𝑣1 𝑣𝑣2
10
JMoon KAIST EE
Principal Component Analysis (PCA):
Eigenvectors and Eigenvalues
Main Concepts: Eigenvector describes the angle or direction while eigenvalue quantifies how
large the variance is in that direction
3 PCs take up 98%
of the total
variance
∑3𝑖𝑖=1 𝜆𝜆𝑖𝑖
= .98
∑𝐾𝐾
𝑖𝑖=1 𝜆𝜆𝑖𝑖
Sushmitha Pulagam
11
JMoon KAIST EE
Principal Component Analysis (PCA):
Conceptually we
• Identify the direction of PCs
• Measure the variance along each PC
• Just keep high variance PCs
As more features are dropped via PCA, the overfitting issue disappears
12
JMoon KAIST EE
Non-Negative Matrix Factorization
• NMF is a non-supervised data decomposition technique.
• NMF automatically extracts sparse and meaningful features from a set of nonnegative data
vectors, but there is no guarantee of algorithm convergence due to lack of global minimum.
• NMF approximates a large matrix 𝑽𝑽 with sparse matrices such that 𝑽𝑽 ≈ 𝑾𝑾𝑾𝑾.
tall wide
Think of columns of V
as individual data
samples.
𝐾𝐾 ≪ 𝐹𝐹, 𝑁𝑁
Every one of these basis or patterns coefficients for combining there are only a
data samples can be data matrix or basic components basic components small number of
approximated by some basic patterns
linear combination of a
few basic patterns or
components.
black: +1
white: 0
elements of W and H are non-negative
(nonnegativity induces sparsity and allows
parts-based representations)
Essid & Ozerov
A column of V is a linear combination of columns of W
Each column n of H tells how to combine the basic patterns (columns of W) to match a specific column n of V 13
JMoon KAIST EE
Non-Negative Matrix Factorization
Let’s flatten each image into a vector of pixels (2D pixels one long 1D pixels)
14
JMoon KAIST EE
Non-Negative Matrix Factorization
each column of V is a face elements in a column of H tell how to combine the basic facial features
(columns of W) to match a specific face
each column here describes how the basic facial
features (columns of W) are combined to estimate
the original image in the matching column of V;
the darkness level represents the weight
columns of W represent some basic facial features,
different combinations of which approximate
individual facial images in columns of V
Essid & Ozerov
15
JMoon KAIST EE
MNIST Dataset
Select images from MNIST Dataset
The MNIST (Modified National Institute of Standards and Technology)
dataset is a large database of handwritten digits commonly used for
training various image processing systems. It contains 60,000 training
images and 10,000 testing images.
16
JMoon KAIST EE
Non-Negative Matrix Factorization
A. Müller, Sarah Guido
each image (flattened version) here represents a column of W
17
JMoon KAIST EE
Non-Negative Matrix Factorization
How NMF components are combined to approximate MNIST images; a set of
coefficients taken from a column of H are used to approximate a specific data
image.
A. Müller, Sarah Guido
18
JMoon KAIST EE
Non-Negative Matrix Factorization
15 basic components; each component images formed by combining components in
image is from each column of W the left (these images apply a heavy weight on
component 3, the left-looking facial feature)
images formed by combining components in
the left (these images apply a heavy weight on
component 7, the right-looking facial feature)
A. Müller, Sarah Guido
19
JMoon KAIST EE
NMF Applications
• Feature learning
– Learn NMF on training dataset
Starting from some initial nonnegative values, 𝑾𝑾 and 𝑯𝑯 are iteratively modified
until the difference between 𝑽𝑽train and 𝑾𝑾𝑯𝑯 becomes small and stable)
– Use 𝑾𝑾 to get 𝒉𝒉𝑛𝑛 for a test sample 𝒗𝒗𝑛𝑛 (from the relationship 𝒗𝒗𝑛𝑛 ≈ 𝑾𝑾𝒉𝒉𝑛𝑛 )
– Now 𝒉𝒉𝑛𝑛 represents 𝒗𝒗𝑛𝑛 with a reduced dimension (dimension reduction
from F to K)
20
JMoon KAIST EE
Dimensionality Reduction with t-SNE
• t-distributed Stochastic Neighbor Embedding (t-SNE) is one of the techniques to visualize high-dimensional data.
• t-SNE is good at grouping samples based on the local structure of the data.
• t-SNE can be slow and computationally expensive.
• Given data points in high dimension, find the pair-wise similarities (Gaussian joint probabilities). A same number of random data
points are then created at some low-dimensional (like dimension 2 or 3) target space. These artificial data points are iteratively
altered (trained) so that their joint probabilities (in low dimension) become similar to those obtained in the original high
dimension. In this way a low dimensional representation that is best for visualization is learned. Let’s not worry about understanding details.
21
JMoon KAIST EE
What is One-Hot-Encoding?
• A way to convert categorical (e.g., red, green, blue) data to numbers
• Many machine learning algorithms cannot operate on label data directly; they require all input
variables and output variables to be numeric
• Simple integer ordinal encoding (e.g., red=1, green=2, blue=3) is not appropriate when there is
no natural ordinal relationship. What makes you say blue is large than green?
• For inherently categorical variables, ordinal encoding imposes an ordinal relationship where no
such relationship may exist; this can mislead the model and a one-hot encoding should be used
instead.
For each new category, simply introduce enough new features such that turning “on” just a single new feature
(with binary 1) would be able to represent a given data sample.
data Category: color data feature: feature: feature:
sample sample # red yellow green
#
1 red 1 1 0 0
2 red 2 1 0 0
3 yellow 3 0 1 0
4 green 4 0 0 1
5 yellow 5 0 1 0
22
JMoon KAIST EE
What is One-Hot-Encoding?
• A way to convert categorical (e.g., red, green, blue) data to numbers
• Many machine learning algorithms cannot operate on label data directly; they require all input
variables and output variables to be numeric
• Simple integer ordinal encoding (e.g., red=1, green=2, blue=3) is not appropriate when there is
no natural ordinal relationship. What makes you say blue is large than green?
• For inherently categorical variables, ordinal encoding imposes an ordinal relationship where no
such relationship may exist; this can mislead the model and a one-hot encoding should be used
instead.
For each new category, simply introduce enough new features such that turning “on” just a single new feature
(with binary 1) would be able to represent a given data sample.
data Category: color data feature: feature: feature:
sample sample # red yellow green
#
1 red 1 1 0 0
2 red 2 1 0 0
3 yellow 3 0 1 0
4 green 4 0 0 1
5 yellow 5 0 1 0
nothing but binary words with just a single position having 1
23