University of Science, VNU-HCM

Faculty of Information Technology

Môn Cơ Sở Trí Tuệ Nhân Tạo

Introduction to Data Science Course

Data Preprocessing

Le Ngoc Thanh
lnthanh@[Link]
Department of Computer Science

Ho Chi Minh City

 Khi tham gia dự án phải có đc dữ liệu CRAWL => thử preProcess xem có gì khác vs DL mẫu đã xử lý
Contents
◎ Why need to preprocess data?
◎ Data cleaning
◎ Data integration
◎ Data reduction
◎ Data transformation

fit@hcmus
 Data
◎ Attribute (Key) - Value
◎ Data types
○ numeric, categorical
○ static, dynamic (time)
◎ Other data types
○ Distributed data
○ Text data
○ Web data, metadata
○ Pictures, audio / video
○ .... speech to text

fit@hcmus
 Data quality
◎ Missing, incomplete: missing attribute value, missing attributes
of interest, or only contains integrated data => thiếu ô, thiếu trường, thiếu cột, thiếu dòng
(sample)
○ Example : age, weight = “ ”
◎ Noise: contain errors or outliers + errors: sai.
+ outliers: biến tồn tại nhưng ko đại diện cho đối tượng đag lm việc
○ Example: salary =“-100 000”
◎ Conflict: there is inconsistency in the code or in the name
○ Example: age =42 , birth = 03/07/1997; US=USA?

fit@hcmus
 Consequences of data quality
◎ The right decision must be based on accurate data
○ For example, duplication or lack of data can lead to inaccurate
statistics, or even misleading.
◎ Data warehouse needs consistent integration of quality
data

"Poor quality data -> not good exploitation"

poor to rich :
1. poor chỗ nào
2. giair quyết nó như thế nào

fit@hcmus
 Solutions? (1/2)
=> xử lý thiếu, nhiễu và conflict

=> khi tổng hợp có thể xảy ra conflict

=> để phù hợp với mô hình

=> có những chiều ko qtrong thì tốn

bộ nhớ và thời gian
=> nén dữ liệu

fit@hcmus
 Solutions? (2/2)
◎ Data Cleaning
○ Fill in missing values, eliminate noise data, identify and eliminate discrepancies,
noise data, and resolve conflicting data
◎ Data Intergration
○ Synthesize, integrate DL from many databases, different files.
◎ Data Transformation
○ Aggregation.
◎ Data Reduction
○ Reduce the data size but ensure analytical results.

fit@hcmus
 Contents
◎ Why need to prepare data?
◎ Data cleaning => thiếu giá trị (ô),
=> thiếu hàng (sample): bài toán ktra nước biển có ô nhiễm ko

◎ Data integration
=> thiếu cột: thiếu đặc trưng (attribute/ feature)

◎ Data reduction
◎ Data transformation

fit@hcmus
 Data cleaning
◎ Data cleaning is the most important task
◎ Data cleaning is the process:
○ Fill in the missing values điền dựa vào cột, dựa vào dòng, dùng mô hình học máy để điền => điền THIẾU Ô
○ Identify and eliminate noise data
○ Resolve conflicting data

fit@hcmus
 => Đánh dấu lại những gì cta đã điền + Cách 1: cách đơn giản nhất là Xóa đi => gây mất DL qtrong or thú vị => cân nhắc
+ Cách 2: Thu thập lại => có đc DL chất lượng nhưng mất chi phí làm lại và có thể

Fill the missing value(1/2) ko lm đc. Nếu ko lm đc thì đến bước tiếp theo
+ Cách 3: Điền
1. Ít tác động đến DL nhất: diền Null, None, ko bk,...=> KQ đ/giácó thể bị ảnh hưởng
2. Dùng kĩ năng thống kê để điền (mean, median, mode..). Nếu vẫn ko ổn thì ta gom
◎ Delete missing items: nhóm DL lại (clustering), mẫu DL rơi vào nhóm nào thì dùng gtri TB của nhóm đó.
Hoặc là dùng ML (từ DL đã biét tôi đi tìm giá trị còn thiếu)
=> TÂT CẢ NHỮNG GÌ ĐÃ LÀM ĐỀU LÀ SAI
○ Commonly used when class labels are missing (in classification)
○ Ease, but not efficiency, especially when the ratio of missing values is
high.
◎ Fill in missing values manually: tasteless and not feasible
◎ Fill missing values automatically:
○ Replaced by a common constant. For example, "don't know". Can
become new class in data
THIẾU CỘT: why know: try and error

fit@hcmus
 Fill the missing value (2/2)
◎ Fill missing values automatically:
○ Replaced with the property's mean
○ Replaced with the property’s mean in a class
○ Replace with the most likely value: infer from a Bayesian formula,
decision tree or EM algorithm (Expectation Maximization)

fit@hcmus
 Data cleaning
◎ Data cleaning is the most important task
◎ Data cleaning is the process:
○ Fill in the missing values
○ Identify and eliminate noise data
○ Resolve conflicting data

fit@hcmus
 Noise reduction
◎ The basic methods of noise reduction:
○ Binning method:
◉ Sort and divide data into equal-width or equal-depth bins
◉ Noise reduction by mean, median, margin, ...
○ Clustering method:
◉ Detect and remove outliers
○ Regression method:
◉ Fit data into the regression function

fit@hcmus
 Noise reduction– Binning (1/4)
◎ Binning method
○ Divide data into equal-width bins:
◉ Divide the range of values into N about the same size
◉ The width of each interval = (maximum value - minimum value) / N
○ Divide data into equal-depth bins:
◉ Divide the range of values into N ranges that each contain approximately the
same number of samples

fit@hcmus
 Noise reduction– Binning (2/4)
◎ Example about equal-width:
The temperature value with N = 7:
64 65 68 69 70 71 72 72 75 75 80 81 83 85

Count

4
2 2 2 0 2 2

[64,67) [67,70) [70,73) [73,76) [76,79) [79,82) [82,85]

Left Bound <= value < Right Bound
s in to N in te rvals.
Divide the range o
f v a lu e
lu e - min imu m v a lue) / N.
a c h in te rv a l = (m aximum va
fit@hcmus The width of e
 Noise reduction– Binning (3/4)
◎ But not good for skewed data

Count

[0 – 200,000) … …. [1,800,000 –
2,000,000]
Salary in the Company
fit@hcmus
 Noise reduction– Binning(4/4)
◎ Example about equal-depth:
The temperature value with N = 4:
64 65 68 69 70 71 72 72 75 75 80 81 83 85
Count

4 4 4
2

[64 .. .. .. .. 69] [70 .. 72] [73 .. .. .. .. .. .. .. .. 81] [83 .. 85]

Depth = 4, except for the last bin

Divide the range of values into N ranges that
each contain approximately the same number of
fit@hcmus
samples
 Noise reduction with split bins
◎ Sorted prices:
4, 8, 15, 21, 21, 24, 25, 28, 34
◎ Divide data into an equal-depth bins with N = 3
○ Bin 1: 4, 8, 15
○ Bin 2: 21, 21, 24
○ Bin 3: 25, 28, 34
→ What to do with the split bins?

fit@hcmus
 Noise reduction with split bins
○ Bin 1: 4, 8, 15
○ Bin 2: 21, 21, 24
○ Bin 3: 25, 28, 34 Smoothing by median:
- Bin 1: 8, 8, 8
- Bin 2: 21, 21, 21
- Bin 3: 28, 28, 28

Smoothing by mean: Smoothing by margin:

- Bin 1: 9, 9, 9 - Bin 1: 4, 4, 15
- Bin 2: 22, 22, 22 - Bin 2: 21, 21, 24
- Bin 3: 29, 29, 29 - Bin 3: 25, 25, 34

fit@hcmus
 Exercises
◎ Prices :
15, 17, 19, 25, 29, 31, 33, 41, 42, 45, 45, 47, 52, 52, 64
◎ Use the binning method with equal-width and equal-depth with
four bins:
○ Calculate the value of the bin according to the median smoothing.
○ Calculate the value of the bin according to the margin smoothing.
○ Calculate the value of the bin according to the mean smoothing.
○ Give some comments on results.

fit@hcmus
 Noise reduction?
◎ The basic methods of noise reduction:
○ Binning method:
◉ Sort and divide data into equal-width or equal-depth bins
◉ Noise reduction by mean, median, margin, ...
○ Clustering method:
◉ Detect and remove outliers
○ Regression method:
◉ Fit data into the regression function

fit@hcmus
 Noise reduction – clustering

fit@hcmus
 Noise reduction – regression

Y1

Y1’ y=x+1

X1 x

fit@hcmus
 Data cleaning
◎ Data cleaning is the most important task
◎ Data cleaning is the process:
○ Fill in the missing values
○ Identify and eliminate noise data
○ Resolve conflicting data

fit@hcmus
 Resolve conflicts
◎ How to handle conflicting data?
◎ Give examples of each conflict resolution method.

fit@hcmus
 Contents
◎ Why need to prepare data?
◎ Data cleaning
◎ Data integration
◎ Data reduction
◎ Data transformation

fit@hcmus
 Data Integration
◎ Select and aggregate data from many different sources
into one database
◎ What problems occur when selecting and aggregating
data?

fit@hcmus
 Data integration process (1/4)
◎ Process:
○ Select only required data for the data mining process.
○ Matches the data schema
○ Eliminate redundant and duplicate data
○ Detect and resolve data inconsistencies

fit@hcmus
 Data integration process (2/4)
◎ Schema Matching
○ Entity recognition problem
◉ How do entities from multiple data sources become relevant
◉ US=USA; customer_id = cust_number
○ Metadata

fit@hcmus
 Data integration process (3/4)
◎ Eliminate redundant and duplicated data
○ An attribute is redundant if it can be inferred from other properties
○ The same property can have multiple names in different databases
○ Some records in the data are repeated
○ Use correlation analysis
◉ r=0: X and Y are not correlated
◉ r>0: positive correlation. Xá«Yá
◉ r<0: negative correlation . Xâ« Y á

fit@hcmus
 Data integration process (4/4)
◎ Resolve inconsistencies in data
○ For example, weight is measured in kilograms or pounds
○ Define standards and mapping based on metadata

fit@hcmus
 Contents
◎ Why need to prepare data?
◎ Data cleaning
◎ Data integration
◎ Data reduction
◎ Data transformation

fit@hcmus
 Data reduction
◎ The data may be too large for some data mining applications:
time consuming.
◎ Data reduction is the process of reducing data (size) so that
the same (or almost the same) analysis result is obtained.

fit@hcmus
 Methods of data reduction
◎ Methods:
○ Aggregation
○ Dimensionality reduction
○ Data compression
○ Numerosity reduction
○ Discretization and Concept hierarchies

fit@hcmus
 Data reduction – Aggregation (1/3)
◎ Aggregation
○ Combination of 2 or more attributes (object) into 1 attribute (object)
◉ Example: cities integrated into regions, regions and water, …
○ Aggregate low-level data into high-level data:
◉ Decrease data set size: reduce the number of attributes
◉ Increase the interestingness of the sample

fit@hcmus
 Data reduction – Aggregation (2/3)

fit@hcmus
 Data reduction – Aggregation (3/3)

fit@hcmus
 Data reduction – Dimensionality reduction (1/6)
◎ Dimensionality reduction
○ Feature selection (subset of attributes)
◉ Choose m from n attributes
◉ Remove irrelevant, redundant attributes
○ How to define irrelevant attributes?
◉ Statistics
◉ Information gain

fit@hcmus
 Data reduction – Dimensionality reduction (2/6)
◎ How to reduce the data dimension?
○ Brute Force
◉ There are 2d attribute subsets of d attributes
◉ Computational complexity is too high
○ Heuristic method
◉ Stepwise forward selection
◉ Stepwise backward elimitation
◉ Combine two methods
◉ Inductive decision tree

fit@hcmus
 Data reduction – Dimensionality reduction (3/6)
◎ Heuristic - Stepwise forward
○ Step 1: choose the best single attribute
○ Step 2: Choose the best attribute from the rest,...
◎ Example with initial attribute set:
{A1,A2,A3,A4,A5,A6}
○ Result ={}
◉ S1: Result = {A1}
◉ S2: Result = {A1,A4}
◉ S3: Result = {A1,A4,A6}

fit@hcmus
 Data reduction – Dimensionality reduction (4/6)
◎ Heuristic - Stepwise backward
○ Step 1: removes the worst single attribute
○ Step 2: continues to remove the worst of the remaining attributes, …
◎ Example with initial attribute set:
{A1,A2,A3,A4,A5,A6}
○ Result ={A1,A2,A3,A4,A5,A6}
◉ S1: Result = {A1,A3,A4,A5,A6}
◉ S2: Result = {A1,A4,A5,A6}
◉ S3: Result = {A1,A4, A6}

fit@hcmus
 Data reduction – Dimensionality reduction (5/6)
◎ Heuristic – Combine Forward and Backward
○ Step 1: select the best single attribute and the worst single attribute type
○ Continue to choose the best attribute and the worst attribute type among the rest, …
◎ Example with initial attribute set: {A1,A2,A3,A4,A5,A6}
○ Result = {A1,A2,A3,A4,A5,A6}
◉ S1: Result = {A1,A3,A4,A5,A6}
◉ S2: Result = {A1,A4,A5,A6}
◉ S3: Result = {A1,A4, A6}

fit@hcmus
 Data reduction – Dimensionality reduction (6/6)
◎ Heuristic – Inductive decision tree
○ Step 1: build decision tree
○ Step 2: removes any properties that are not present on the tree
◎ Example with initial attribute set:
{A1,A2,A3,A4,A5,A6}
Þ Result = {A1, A4, A6} A4 ?

A1? A6?

Class 2 Class 1 Class 2

Class 1
fit@hcmus
 Data reduction – Compression
◎ Data Compression:
○ Encrypt or transform data
○ Lossless compression
◉ Data can be recovered
○ Lossy compression
◉ Data cannot be fully recovered
○ Using wavelet transforms, principal component analysis (PCA), ...

fit@hcmus
 Data reduction – Numerosity reduction
◎ Numerosity reduction: selects a different representation of
the data ("less than")
◎ Some methods:
○ Parameter method:
◉ Use a mathematical model to store parameters
◉ Regression model and log-linear
○ Non-parametric method:
◉ Do not use a mathematical model but save the reduced representation
◉ Graphs, grouping, sampling

fit@hcmus
 Data reduction – Numerosity reduction
◎ Linear regression:Y = a + b X
◎ Multi linear regression: Y = b0 + b1 X1 + b2 X2
◎ Log-linear model:
○ Probaility: p(a, b, c, d) = aab bac cad dbcd

fit@hcmus
 Data reduction – Numerosity reduction
◎ Histogram
○ Common methods for data reduction
○ Divide the data into bins and the height of the column is the number
of objects in each bin. Store only the average of each bin.
○ The shape of the chart depends on the number of bins

fit@hcmus
 Data reduction – Numerosity reduction
◎ Clustering
○ Divide data into groups and save group representations.
○ Very effective if the data is grouped but vice versa when the data is scattered
○ Lots of clustering algorithms.

fit@hcmus
 Data reduction – Numerosity reduction
◎ Sampling
○ Use a much smaller random sample set instead of large data set.
○ Simple random sample without replacement (SRSWOR)
○ Simple random sample with replacement (SRSWR)
○ Group / hierarchical sampling method

fit@hcmus
 Data reduction – Numerosity reduction

W O R
SRS le random
p t
(sim le withou
samp ment)
p la ce
re

SRSW
R

Raw Data
fit@hcmus
 Data reduction – Numerosity reduction

Raw Data Cluster/Stratified Sample

fit@hcmus
 Data reduction – Discretization and Concept hierarchies

◎ Discretization:
○ Converts the property value domain (contiguous) by dividing the
value domain into intervals.
○ Store labels of ranges instead of actual values
○ Suitable for continuous numeric data.
○ Methods: binning, chart analysis, grouping, discrete by entropy,
natural segmentation.

fit@hcmus
 Data reduction – Discretization and Concept hierarchies

◎ Concept hierarchies:
○ Gather and replace a low-level concept with a higher-level concept.
○ Suitable for non-numeric data: create a hierarchy.

fit@hcmus
fit@hcmus
 Data reduction – Discretization and Concept hierarchies

◎ Example:
○ Converts the logical value to 1.0
○ Converts a date value to a number
○ Converts columns with large numeric values into a set of values in a smaller range,
for example dividing them by a certain factor.
○ Group of values has the same semantics as: Activity before August Revolution is
group 1; from 01/08/45 - 31/06/54; group 2; from 01/07/54 - 30/4/75 is group 3, ...
○ Substitute the value of age into young, middle-aged, old

fit@hcmus
 Contents
◎ Why need to prepare data?
◎ Data cleaning
◎ Data integration
◎ Data reduction
◎ Data transformation

fit@hcmus
 Data transformation
◎ Data transformation: convert data into a form that is
suitable and convenient for algorithms
◎ Data transformation process :
○ Smoothing
○ Aggregation
○ Generalization
○ Normalization
○ Attribute construction

fit@hcmus
 Data transformation process
◎ Smoothing: the process of removing noise from the data.
◎ Integration: summarizing or integrating data.
◎ Generalization: replacing low-level concepts with high-level
concepts.
◎ Normalization: attribute data should be returned to a small
range of values like 0 to 1.
◎ Attribute construction: new properties are created and added
to a given set of properties

fit@hcmus
 Conclusion
◎ Data is often missing, noisy, inconsistent, and
multidimensional.
◎ Good data is the key to creating reliable and valid models.
◎ Data preparation includes the following processes:
○ Cleaning
○ Selection
○ Reduction
○ Transformation

fit@hcmus
 Exercises
◎ Why is preparing data so urgent and time-consuming?
◎ How to solve the problem of missing values in database
records?
◎ Assuming the database has Age attribute with the values in
the records (ascending): 13, 15, 16, 16, 19, 20, 20, 21, 22, 22,
25, 25, 25, 25, 30, 33, 33, 35, 35, 35, 35, 36, 40, 45, 46, 52,
70
○ Denoise data by mean of bins with the number of bins n = 4. Explain the
effectiveness of this technique with the above data.
○ Plot the equal-width histogram with the width = 10

fit@hcmus
 Exercises
◎ Why do we need to select / integrate data? Please describe
the data selection process.
◎ Why need to data reduction? Can data reduction process lose
information? If yes, please state how to fix it.
◎ Learn about the data transformation processes. Give
examples for each direction.

fit@hcmus
fit@hcmus