Artificial Intelligence for Biotechnology
Data Preprocessing & Normalization
Prof. Dr. Dominik Grimm
TUM Campus Straubing for Biotechnology and Sustainability
Data Preprocessing & Normalization
Learning objectives of this lecture
• Sample Matching and Filtering
• Imputing missing values
• Data normalization
• Encoding of features
• Transformation of features
Matching Samples
» The order of the samples in the feature matrix and the label vector might differ → match samples
» There might be samples for which no labels are known → filter samples
Labels Features
S1 S10
S2 S2
S7
Sample IDs
Sample IDs
S4 S1
S5 S12
S6
S7 S3
S8 S9
S11
S10 S4
S11
S12 S5
Prof. Dr. Dominik Grimm (TUMCS) 3
Matching Samples
Matching Samples using NumPy
import numpy as np b
a[:,[Link]]
s4 s3 s6 s1
a = [Link](["s1","s2","s3"])
s1 False False False True
b = [Link](["s4","s3","s6","s1"])
s2 False False False False
truth_table = (a[:,[Link]]==b) s3 False True False False
Tuple with indices of a and b
ind = [Link](truth_table==True) (array([0, 2]), array([3, 1]))
Indices of a Indices of b
print(a[ind[0]]) #→ array(['s1', 's3'], dtype='<U2')
print(b[ind[1]]) #→ array(['s1', 's3'], dtype='<U2')
Prof. Dr. Dominik Grimm (TUMCS) 4
Missing Data
There might be samples with missing features → filter samples or impute missing values
Labels Features
S1 S10
S2 S2
S7
Sample IDs
Sample IDs
S4 S1
S5 S12
S6
S7 S3
S8 S9
S11
S10 S4
S11
S12 S5
Missing values might have the numpy type [Link], but could also have other
values, such as “?” or just an empty string
Prof. Dr. Dominik Grimm (TUMCS) 5
Missing Data
Remove Samples
sample_ids_to_remove = [Link]([„s2“, „s4“])
#get truth array
truth_table = (sample_ids[:,[Link]] == sample_ids_to_remove)
#get indicies where truth matrix is True
ind = [Link](truth_table==True)
y = [Link](y, ind[0], axis=0) #remove labels of samples not needed
X = [Link](X, ind[0], axis=0) #remove all rows of samples not needed
Prof. Dr. Dominik Grimm (TUMCS) 6
Imputing Missing Data
Simple Imputation of Categorical or Binary Features
Feature 𝑥𝑗 Imputed Feature 𝑥𝑗
1 1
0 0
𝑖
𝑥𝑗
3
NAN
Impute the missing feature 𝑥𝑗 for sample 𝑖 using the most
0
frequent value along the feature column 𝑗
0 0
0 0
0 0
7
𝑥𝑗 NAN 0
1 1
Prof. Dr. Dominik Grimm (TUMCS) 7
Imputing Missing Data
Simple Imputation of Continuous Features: Mean or Median Imputation
Feature 𝑥𝑗 Mean-Imputed Feature 𝑥𝑗
2.6 2.6
7.2 7.2
𝑖
𝑥𝑗
3
NAN
Impute the missing feature 𝑥𝑗 for sample 𝑖 using the mean
4.3
or median value along the feature column 𝑗
2.3 2.3
6.2 6.2
3.2 3.2
7
𝑥𝑗 NAN 4.3
4.2 4.2
Prof. Dr. Dominik Grimm (TUMCS) 8
Imputing Missing Data
Simple imputation using sklearn
X = [Link]([[1,[Link],3],
[[Link],5,2],
[0,6,[Link]]])
#Import Simple Imputer Class
from [Link] import SimpleImputer imputer = SimpleImputer(
missing_values=[Link],
imputer = SimpleImputer( strategy=‘most_frequent')
missing_values=[Link], X_im = imputer.fit_transform(X)
strategy='mean') print(X_im)
X_im = imputer.fit_transform(X)
print(X_im) [[1., 5, 3. ]
[0, 5., 2. ]
[[1., 5.5, 3. ] [0., 6., 2]]
[0.5, 5., 2. ]
[0., 6., 2.5]]
Prof. Dr. Dominik Grimm (TUMCS) 9
Imputing Missing Data
Imputation using k-Nearest-Neighbor
Idea: Approximate missing values by looking at its closest neighbor(s).
X = [Link]([[1,[Link],3],
[[Link],5,2],
[0,6,[Link]]])
#Import Simple Imputer Class
from [Link] import KNNImputer
imputer = KNNImputer(missing_values=[Link])
X_im = imputer.fit_transform(X)
print(X_im)
[[1., 5.5, 3. ]
[0.5, 5., 2. ]
[0., 6., 2.5]]
Prof. Dr. Dominik Grimm (TUMCS) 10
Imputing Missing Data
Example
75% Random Noise Mean Imputation kNN Imputation (k=1)
Prof. Dr. Dominik Grimm (TUMCS) 11
Imputing Missing Data
Random Noise Mean Imputation kNN Imputation (k=1) Original
20%
40%
75%
Prof. Dr. Dominik Grimm (TUMCS) 12
Imputing Missing Data
Important: Only use the training data to estimate the values for imputation.
Do not compute these values on the test data again. Apply the values
estimated on the training data on the test data!
Important: Never impute your labels!
Prof. Dr. Dominik Grimm (TUMCS) 13
Categorical Features
Categorical data must be separated into:
Categorial Data
Ordinal Data Nominal Data
Categorial values that Categorial values that
can be sorted or ordered don’t imply any order
• Grades • Color
• T-Shirt Size (S, M, L) • Car model
• Interval Scale: • Style of haircut
Child (0-12y) • Brand name
Teenager (13-19y)
Prof. Dr. Dominik Grimm (TUMCS) 14
Categorical Features
Ordinal data
Ordinal features must be encoded into integers. However, the ordering is important and must
be meaningful!
L Meaningful encoding
S=1
XL M=2 Integers increase with size of shirt
M L=3
XL = 4
S
Get unique
[ L, XL, M, S ]
elements
L
Bad encoding
L S=4 Why has a small shirt a larger integer
M=3 than a large shirt? Here, a shirt of size S
XL would weigh four times higher than a
L=1 shirt of size L
L XL = 2
Prof. Dr. Dominik Grimm (TUMCS) 15
Categorical Features
Nominal data
Nominal data has no particular order. To encode such data one could create dummy variables
using a technique which is called one-hot-encoding. Here, one would convert a single feature
into several binary features:
X = [Link]([[1,3,"green"],
red green blue [2,5,"red"],
[0,6,"green"]])
blue 0 0 1
from [Link] import OneHotEncoder
green 0 1 0
red 1 0 0 #categorical_features is a list of indices
#which columns have to be one hot encoded
blue 0 0 1 encoder = OneHotEncoder()
Xe= encoder.fit_transform(X[:,2:])
green 0 1 0 print([Link]())
red 1 0 0 [[1. 0.]
[0. 1.]
red 1 0 0 [1. 0.]]
Prof. Dr. Dominik Grimm (TUMCS) 16
Feature Standardization and Normalization
Feature scaling (standardization or normalization) is a crucial step before running
any machine learning model. Most of the machine learning algorithms are not scale
invariant! Feature scaling also leads to more stable parameter estimates and faster
convergence of gradient based optimization techniques.
Example
Let’s assume we have two features. The first one has a range between 1 and 20 and
the second one has a range between 100 and 20000.
» When computing the l2-loss (squared-error) for linear regression the error is
dominated by the second feature, because the errors are larger than on the first
feature.
» When using kNN the Euclidean distance will be dominated by the second feature.
Distances between samples might become arbitrarily large.
Prof. Dr. Dominik Grimm (TUMCS) 17
Feature Standardization and Normalization
Min-Max Scaling
Min-Max Scaling rescales each individual feature 𝑗 to a range between 0 and 1:
𝑖
𝑖
𝑥𝑗 − min 𝒙𝑗
𝑥𝑗 =
max 𝒙𝑗 − min 𝒙𝑗
Largest value in feature column 𝒋 Smallest value in feature column 𝒋
from [Link] import MinMaxScaler
minmax = MinMaxScaler()
X_train = minmax.fit_transform(X_train) Compute transform on training data
X_test = [Link](X_test) Apply same transformation (min, max from
training data) on test data!
Prof. Dr. Dominik Grimm (TUMCS) 18
Feature Standardization and Normalization
Z-Score (Zero Mean Unit Variance)
With this standardization we center the feature j at mean 0 with a standard deviation of 1. This
does not change the relative difference of the feature between the samples. This
standardization is useful for learning weights in logistic regression or SVM:
𝑖 Mean of feature 𝒋
𝑖
𝑥𝑗 − 𝜇𝑥𝑗
𝑥𝑗 =
𝜎𝑥𝑗 Standard deviation of feature 𝒋
from [Link] import StandardScaler
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train) Compute transform on training data
X_test = [Link](X_test) Apply same transformation (mean, std from
training data) on test data!
Prof. Dr. Dominik Grimm (TUMCS) 19
Feature Standardization and Normalization
Example
Prof. Dr. Dominik Grimm (TUMCS) 20
Feature Standardization and Normalization
» Standardization and Normalization are important before running any
machine learning model
» It helps to mediate the effect of different scales of features
» It helps to speed up the training and leads to faster convergence of the
algorithms
Important: Values for the minimum, maximum, mean or standard deviation are
always estimated on the training data. The exact same parameters are then used
to transform the test data. Do not compute these values on the test data again!
Prof. Dr. Dominik Grimm (TUMCS) 21
Objectives: What should I know?
» How to filter or match samples
» What is imputation and how to import missing values
» How to encode categorial or text-based features
» How to normalize or standardize features
Prof. Dr. Dominik Grimm (TUMCS) 22
Thanks for your attention!
Homepage & Contact Details
Prof. Dr. Dominik Grimm
Petersgasse 18
Raum 00.027
[Link]@[Link]
[Link]
@dg_grimm
Prof. Dr. Dominik Grimm (TUMCS) 23