2.

Data Pre-processing

C MT 307 A ppl i ed Mach i n e L earni ng

Yu h u a L i
l i y180@cardi [Link] .u k

CM307 1
Logical Process of Machine Learning

EDA Data
Data Data modelling / Model
collection preprocessing machine learning application

Model New/unseen
Unsatisfied evaluation data

CM307 2
Contents
• Introduction
• Data inspection
• Transformation
• Dealing with missing data
• Encoding categorical features
• Imbalanced data
• Data reduction

CM307 3
Introduction
• Why is data pre-processing Important?
◦ No quality data, no quality mining results!
◦ Quality decisions must be based on quality data
e.g., duplicate or missing data may cause incorrect
or even misleading statistics.
◦ Data preparation, cleaning, and transformation comprises the majority of the
work in a machine learning application.

CM307 4
Introduction
• Data Validation
◦ Data in the real world is dirty
◦ Incompleteness
§ missing attribute values, lack of certain attributes of interest, or containing only aggregate
data. E.g. Postcode = “”
◦ Noise
§ Data may contain erroneous or anomalous values. These are referred to as outliers. E.g.
age = -2
◦ Inconsistency
§ Sometimes data contains discrepancies.
e.g., Age=“52” Birthday=“01/01/1991”

CM307 5
Noisy Data
• Noise: random error or variance in a measured variable.

• Incorrect attribute values may due to

◦ faulty data collection instruments
◦ data entry problems
◦ data transmission problems
◦ etc

• Other data problems which requires data cleaning

◦ duplicate records, incomplete data, inconsistent data

6
Contents
• Introduction
• Data inspection
• Transformation
• Dealing with missing data
• Encoding categorical features
• Imbalanced data
• Data reduction

CM307 7
Data Inspection - Exploratory data analysis
• Scatter plot
◦ Provides a first look at bivariate data to see clusters of points, outliers, etc
◦ Each pair of values is treated as a pair of coordinates and plotted as points in the plane

Positively correlated Negative correlated Nonlinear correlated Uncorrelated

Correlation coefficient ϵ [-1, 1],

CM307 8
Data Inspection
• Histogram
◦ Graph display of tabulated frequencies, shown
as bars. x-axis are values, y-axis represent
frequencies 40
35
30
25

◦ It shows what proportion of cases fall into each 20

of several categories 15
10
5
0
10000 30000 50000 70000 90000

◦ The categories are usually specified as non-

overlapping intervals of some variable. The
categories (bars) must be adjacent

CM307 9
 Data Inspection
• Five-number summary of a distribution
◦ Minimum, Q1, Median, Q3, Maximum

• Boxplot
◦ Data is represented with a box
◦ The ends of the box are at the first and third
quartiles, i.e., the height of the box is IQR
◦ The median is marked by a line within the box
◦ Whiskers: two lines outside the box extended to
(Q1 – 1.5*IQR) and (Q3 + 1.5*IQR)
◦ Outliers: points beyond whiskers, plotted
individually

10
Contents
• Introduction
• Data inspection
• Transformation
• Dealing with missing data
• Encoding categorical features
• Imbalanced data
• Data reduction

CM307 11
Data Transformation
• Smoothing: remove noise from data (binning, clustering, regression)
• Scaling: scaled to fall within a small, specified range such as –1.0 to
1.0, 0.0 to 1.0 or N(0, 1)
• Attribute/feature construction
◦ New attributes constructed / added from the given ones
• Aggregation: summarization or aggregation operations apply to data
• Generalization: concept hierarchy climbing
◦ Low level/ primitive/raw data are replace by higher level concepts

CM307 12
Feature Scaling (Data Normalization)
• The range of values of raw data varies widely, scaling makes all features have a same range
◦ Necessary for distance/similarity based ML methods
◦ Faster convergence for gradient descent learning algorithms
◦ Appropriately penalizing coefficients in loss functions involving regularisation

• min-max normalization
◦ For a variable X, 𝑥 ∈ 𝑋, the range of X is normalized from its original range 𝑚𝑖𝑛! , 𝑚𝑎𝑥! to new range 𝑎, 𝑏
𝑏−𝑎
𝑥! = 𝑎 + 𝑥 − 𝑚𝑖𝑛"
𝑚𝑎𝑥" − 𝑚𝑖𝑛"

E.g., Normalising dataset (9, 22, 14, 2, 11) to new range [-1, 1] becomes (-0.3 1.0 0.2 -1.0 -0.1)
[Link] [Link].minmax_scale

• z-score normalization (Standardization)

!
𝑥 − 𝑚𝑒𝑎𝑛"
𝑥 =
𝑠𝑡𝑑"
[Link]

CM307 13
Contents
• Introduction
• Data inspection
• Transformation
• Dealing with missing data
• Encoding categorical features
• Imbalanced data
• Data reduction

CM307 14
Dealing with missing data

Survived Pclass Sex Age SibSp Parch Fare Cabin Embarked

0 3 male 22 1 0 7.25 NaN S
1 1 female 38 1 0 71.2833 C85 C
1 3 female 26 0 0 7.925 NaN S
1 1 female 35 1 0 53.1 C123 S
0 3 Male 35 0 0 8.05 NaN S
0 3 Male NaN 0 0 8.4583 NaN Q
0 1 Male 54 0 0 51.8625 E46 S
0 3 Male 2 3 1 21.075 NaN S
1 3 Female 27 0 2 11.1333 NaN S
1 2 Female 14 1 0 30.0708 NaN C

CM307 15
Dealing with Missing Data
• Elimination
◦ Discard all records for which the values of one or more attributes are missing.
In the case of a supervised data mining analysis, it is essential to eliminate a
record if the value of the target attribute is missing.
• Inspection
◦ Alternatively, one may opt for an inspection of each missing value, carried out
by experts in the application domain, in order to obtain recommendations on
possible substitute values.

CM307 16
Dealing with Missing Data
• Identification
◦ As a third possibility, a conventional value might be used to encode and
identify missing values, making it unnecessary to remove entire records from
the given dataset.
• Missing-data imputation
◦ It is the process of replacing missing data with substituted values
◦ Several criteria exist for the automatic replacement of missing data, although
most of them appear somehow arbitrary. For instance, missing values of an
attribute may be replaced with the mean of the attribute calculated from the
remaining observations.

CM307 17
Dealing with Missing Data
• Get rid of the corresponding
instances. Survived Pclass Sex Age SibSp Parch Fare Cabin Embarked
0 3 male 22 1 0 7.25 NaN S
• Get rid of the whole attribute. 1 1 female 38 1 0 71.2833 C85 C
1 3 female 26 0 0 7.925 NaN S
• Set the values to some value 1 1 female 35 1 0 53.1 C123 NaN
(zero, the mean, the median, 0 3 Male 35 0 0 8.05 NaN S

etc.). 0 3 Male NaN 0 0 8.4583 NaN Q

0 1 Male 54 0 0 51.8625 E46 S
• Use imputation methods. 0 3 Male 2 3 1 21.075 NaN S
1 3 Female 27 0 2 11.1333 NaN S
[Link]: Impute 1 2 Female 14 1 0 30.0708 NaN C

CM307 18
Contents
• Introduction
• Data inspection
• Transformation
• Dealing with missing data
• Encoding categorical features
• Imbalanced data
• Data reduction

CM307 19
Encoding categorical features
• Ordinal
◦ May use label encoding (integer encoding)
[Link] category_encoders.[Link]
Pclass Pclass Degree Degree
Third class 2 BSc 1
First class 0 No degree 0
Third class 2 BSc 1
First class 0 PhD 3
Third class 2 PhD 3
Third class 2 BSc 1
First class 0 MSc 2
Third class 2 No degree 0
Third class 2 MSc 2
Second class 1 BSc 1
CM307 20
Encoding categorical features: nominal
• One hot encoding
◦ Create a new binary feature to represent each of the categories
[Link] category_encoders.one_hot.OneHotEncoder
Sex Embarked
Sex Embarked Female Male C Q S
male S 0 1 0 0 1
female C 1 0 1 0 0
female S 1 0 0 0 1
female S 1 0 0 0 1
Male S 0 1 0 0 1
Male Q 0 1 0 1 0
Male S 0 1 0 0 1
Male S 0 1 0 0 1
Female S 1 0 0 0 1
Female C 1 0 1 0 0
CM307 21
Encoding categorical features: nominal
• Binary Encoding
◦ The categorical feature is first converted integer binary split
into numerical using an ordinal encoder. Degree Degree Degree Deg1 Deg2
Then the numbers are transformed in BSc 1 01 0 1
the binary number. After that binary
No degree 0 00 0 0
value is split into different columns.
BSc 1 01 0 1
PhD 3 11 1 1
◦ Binary encoding works really well when PhD 3 11 1 1
there are a high number of categories.
BSc 1 01 0 1
For example, occupations.
MSc 2 10 1 0
No degree 0 00 0 0
category_encoders.[Link] MSc 2 10 1 0
BSc 1 01 0 1

E.g., a feature with upto 64 categories needs only 6 bits to encode using binary encoding.
CM307 22
Contents
• Introduction
• Data inspection
• Transformation
• Dealing with missing data
• Encoding categorical features
• Imbalanced data
• Data reduction

CM307 23
Data Preprocessing - Imbalanced data
• Imbalanced data refers to classification problems where the class
distribution is not uniform among the classes.
◦ Typically, they are composed by two classes: The majority (negative) class
and the minority (positive) class.
• Oversampling
◦ appends data to the minority class of original data set
e.g., SMOTE (Synthetic Minority Over-sampling Technique)

• Undersampling
◦ removes data from the majority class of the original data set

Imbalanced learning package: imblearn (pip install imbalanced-learn)

CM307 24
Imbalanced Data

b’
a’

CM307 25
Contents
• Introduction
• Data inspection
• Transformation
• Dealing with missing data
• Encoding categorical features
• Imbalanced data
• Data reduction

CM307 26
Data Preprocessing - Data reduction
• Data reduction
◦ Obtains a reduced representation of the data set that is much smaller in
volume but yet produce the same (or almost the same) analytical results
• Date Reduction Criteria
◦ Efficiency
The application of learning algorithm to a dataset smaller than the original one usually means
a shorter computational time.
◦ Accuracy
The accuracy of the models generated represents a critical success factor, it is therefore the
main criterion followed in order to select one class of learning methods over another.
◦ Simplicity
It is important that the models generated by easily translated into simple rules that can be
understood by experts in the application domain.

CM307 27
Data Preprocessing - Data reduction
Ways of Data Reduction
• Feature selection eliminates from the dataset a subset of variables
which are not deemed relevant for the purpose of data mining
activities.
• Instance selection (Pattern selection ) selects a subset of patterns
from the original data that can sufficiently represent the original data
in terms of some specified criteria (e.g., generalization accuracy).
• Data transformation transform original data into a new space so
that the original data can be informatively represented in a lower
dimensional space.
CM307 28
Ways of Data Reduction
x1 x2 x3 xd
X1
X2 X y
X3
Input
System Output

y = f(X)
D = (X1 X2 X3 …… XN) t
X = (x1 x2 x3 …… xd) t
XN

Everything that can be counted does not necessarily count;

everything that counts cannot necessarily be counted.
Albert Einstein

CM307 29
Ways of Data Reduction
x1 x2 x3 xd
X1
X2
X3

XN
Feature
X1 selection
X2
X3

XN

CM307 30
Ways of Data Reduction
x1 x2 x3 xd
X1 Instance
X2
X3
selection
x1 x2 x3 xd

XN

sklearn.feature_selection
X1
X2 Feature
X3 selection
SVM
RVM
RBF y = f(X)
GMM
XN …

CM307 31
Feature selection: Methods
• Can be divided into:
◦ Unsupervised: Make use of unlabelled data only (e.g. remove sparse or low-
variance features, based on their entropy, etc.).
◦ Supervised: Make use of the output labels, and generally are aimed at
removing features that are not relevant or do not help to improve the
performance of the machine learning model.

CM307 32
Supervised Feature Selection Methods
• Supervised feature selection methods can be further split into:
◦ Filter methods: Statistical tests to score each feature.
§ Examples: Chi-squared test, correlation.
◦ Embedded: Learn the most relevant features while the model is being created.
Regularization is the most common technique.
§ Examples: SMLR, LASSO, Ridge Regression
◦ Wrapper: Consider the selection of features as a search problem.
§ Examples: Forward/Backward selection, Recursive Feature Elimination

CM307 33
Instance/Pattern selection

Edge patterns 1

Interior patterns
0.5

ω1

-0.5
ω2

Overlap patterns -1
-1 -0.5 0 0.5 1

Yuhua Li, Liam Maguire (2011)

“Selecting critical patterns based on local geometrical and statistical information,”
IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 6, pp. 1189-1201.

34
Data transformation
Dimensionality reduction

• Principal component analysis

[Link]
[Link]

• Manifold learning
◦ is an approach to non-linear
dimensionality reduction
[Link]
UMAP: [Link]

CM307 35
Machine Learning Model in sklearn
Data

Pre-processing Train test split

Define model
estimator estimator

[Link]() Model training & optimisation [Link]()

[Link]() [Link]()
Model evaluation & selection

CM307 36
Readings
• Chapter 2
Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd
Edition by Aurélien Géron

CM307 37