ML
LECTURE 2
Know your data
Data Preprocessing
Data Preprocessing
Data Preprocessing: An Overview
Major Tasks in Data Preprocessing
Data Cleaning
Data Integration
Data Transformation
Data Reduction
Summary
2
2
Why Preprocess Data?
Welcome to the Real World! Data is Dirty!
But, No quality data, no quality mining results!
Preprocessing is one of the most critical steps in
a data mining process
Why Preprocess Data? Contd.
Measures for data quality: A multidimensional view
Accuracy: correct or wrong, accurate or not
Completeness: not recorded, unavailable, …
Consistency: some modified but some not, dangling, …
Timeliness: timely update?
Believability: how trustable the data are correct?
Interpretability: how easily the data can be understood?
4
Data Preprocessing: Major Tasks
5
5
Data cleaning
Data Integration
Data transformation
Data reduction
Data Preprocessing
Data Preprocessing: An Overview
Data Quality
Major Tasks in Data Preprocessing
Data Cleaning
Data Integration
Data Transformation
Data Reduction
Summary
7
7
Why Data Cleaning?
Data in the real world is dirty. ‘Dirty’ denotes-
Incomplete/missing: lacking attribute values, lacking certain
attributes of interest, or containing only aggregate data
e.g., occupation=“ ”
noisy: containing errors or outliers
e.g., Salary=“-10”
inconsistent: containing discrepancies in codes or names
e.g., Age=“42” Birthday=“03/07/1997”
e.g., Was rating “1,2,3”, now rating “A, B, C”
e.g., discrepancy between duplicate records
Redundant: including everything, some of which are
irrelevant to our task
Duplicate: Data-entry is duplicated.
Dirty Data
Examples of data quality problems:
Noise and outliers
Missing values
Duplicate data Tid Refund Marital
Status
Taxable
Income Cheat
1 Yes Single 125K No
2 No Married 100K No
3 No Single 70K No
4 Yes Married 120K No
A mistake or a millionaire? 5 No Divorced 10000K Yes
6 No NULL 60K No
Missing values 7 Yes Divorced 220K NULL
8 No Single 85K Yes
9 No Married 90K No
Inconsistent duplicate entries
9 No Single 90K No
10
Data Cleaning
Data cleaning tasks
Fill in missing values
Identify outliers/noise and smooth out noisy data
Correct inconsistent data
Remove duplicate/redundant data
Missing Data
Data is not always available
E.g., many tuples have no recorded value for several attributes, such as
customer income in sales data
Missing data may be due to
equipment malfunction
inconsistent with other recorded data and thus deleted
data not entered due to misunderstanding
certain data may not be considered important at the time of entry
not register history or changes of the data
Missing data may need to be inferred
How to Handle Missing Data?
Ignore the tuple: usually done when class label is missing (assuming the
task is classification—not effective in certain cases)
Fill in the missing value manually: tedious + infeasible?
Use a global constant to fill in the missing value: e.g., “unknown”,
a new class?!
Use the attribute mean to fill in the missing value
Use the attribute mean for all samples of the same class to fill in
the missing value: smarter
Use the most probable value to fill in the missing value:
inference-based such as regression, Bayesian formula, decision tree
Noisy Data
What is noise?
- Random error in a measured variable.
Incorrect attribute values may be due to
faulty data collection instruments
data entry problems
data transmission problems
technology limitation
inconsistency in naming convention
How to Handle Noisy Data?
Binning/bucketing
first sort data and partition into (equal-frequency) bins
then one can smooth by bin means, smooth by bin median,
smooth by bin boundaries, etc.
Regression
smooth by fitting the data into regression functions
Clustering
detect and remove outliers
Combined computer and human inspection
detect suspicious values and check by human (e.g., deal with
possible outliers)
Simple Discretization Methods: Binning
Equal-width (distance) partitioning:
It divides the range into N intervals of equal size: uniform grid
if A and B are the lowest and highest values of the attribute, the
width of intervals will be: W = (B-A)/N.
Advantage/disadvantage
The most straightforward
But outliers may dominate presentation
Skewed data is not handled well.
Equal-depth (frequency) partitioning:
It divides the range into N intervals, each containing
approximately same number of samples
Advantage/disadvantage
Good data scaling
Managing categorical attributes can be tricky.
Example
width of intervals will be: W = (B-A)/N
Number of bins,
N=3
W= (215-5)/3
[5~75] = 70
[75~145]
[146~215]
Equal depth/frequency binning: An example
* Sorted data for price (in dollars): 4, 8, 9, 15, 21, 21, 24, 25,
26, 28, 29, 34
* Partition into (equi-depth) bins:
- Bin 1: 4, 8, 9, 15
- Bin 2: 21, 21, 24, 25
- Bin 3: 26, 28, 29, 34
* Smoothing by bin means:
- Bin 1: 9, 9, 9, 9
- Bin 2: 23, 23, 23, 23
- Bin 3: 29, 29, 29, 29
* Smoothing by bin boundaries:
- Bin 1: 4, 4, 4, 15
- Bin 2: 21, 21, 25, 25
- Bin 3: 26, 26, 26, 34
How to Handle Inconsistent Data?
Inconsistent data may come from
Different data sources
Functional dependency violation (e.g., modify some linked data)
Handling Inconsistent Data-
Manual correction using external references
Semi-automatic using various tools
To detect violation of known functional dependencies and data
constraints
To correct redundant data
Data Preprocessing
Data Preprocessing: An Overview
Data Quality
Major Tasks in Data Preprocessing
Data Cleaning
Data Integration
Data Transformation
Data Reduction
Summary
20
20
Data Transformation
√ Smoothing: remove noise from data (binning, clustering,
regression)
√ Aggregation: summarization, data-cube generation
Generalization: concept hierarchy climbing
√ Normalization: scaled to fall within a small, specified range
min-max normalization
z-score normalization
normalization by decimal scaling
√ Attribute/feature construction
New attributes constructed from the given ones
Data Summarization
Summary Statistics
Summary statistics are numbers that summarize properties of
the data
Summarized properties include frequency, location and spread
Examples: location - mean
spread - standard deviation
Most summary statistics can be calculated in a single pass through the
data
Central Tendency: Mean and Median
The mean/average is the most common measure of the
location of a set of points.
However, the mean is very sensitive to outliers.
Thus, the median or a trimmed mean is also commonly
used.
Frequency and Mode
The frequency of an attribute value is the
percentage of time the value occurs in the
data set
For example, given the attribute ‘gender’ and a representative
population of people, the gender ‘female’ occurs about 50% of
the time.
The mode of an attribute is the most frequent attribute value
Data Transformation: Normalization
Data Transformation: Normalization
Min-max normalization
Z-score normalization
Decimal Scaling
min-max normalization
Given Dataset, [A: 8, 10, 15, 20]
Task: Perform min-max normalization on the given data, and calculate normalized values.
Equation
min-max normalization (contd.)
Given Dataset, [A: 8, 10, 15, 20]
Task: Perform min-max normalization on the given data within a range [40, 60], , and calculate
normalized values.
Equation
Z-Score Normalization
Given Dataset, [A: 8, 10, 15, 20]
Task: Perform z-score normalization on the given data, and calculate normalized values.
Equation
max
0ne 0
max
Three 0