Data preprocessing
Data summarization, cleaning, integration and
transformation, Reduction, Discretization and
concept hierarchy generation
Data Mining: Data Preprocessing lecture
notes
Data Preprocessing
Why preprocess the data?
Descriptive data summarization
Data cleaning
Data integration and transformation
Data reduction
Summary
Data Mining: Data Preprocessing lecture notes
Why Data Preprocessing?
Data in the real world is dirty
incomplete: lacking attribute values, lacking
certain attributes of interest, or containing
only aggregate data
e.g., occupation=“ ”
noisy: containing errors or outliers
e.g., Salary=“-10”
inconsistent: containing discrepancies in
codes or names
e.g., Age=“42” Birthday=“03/07/1997”
e.g., Was rating “1,2,3”, now rating “A, B,
C”
e.g., discrepancy between duplicate
Data Mining: Data Preprocessing lecture notes
Why Is Data Dirty?
Incomplete data may come from
“Not applicable” data value when collected
Different considerations between the time when the data
was collected and when it is analyzed.
Human/hardware/software problems
Noisy data (incorrect values) may come from
Faulty data collection instruments
Human or computer error at data entry
Errors in data transmission
Inconsistent data may come from
Different data sources
Functional dependency violation (e.g., modify some linked
data)
Duplicate records also need data cleaning
Data Mining: Data Preprocessing lecture notes
Why Is Data Preprocessing
Important?
No quality data, no quality mining results!
Quality decisions must be based on quality data
e.g., duplicate or missing data may cause incorrect or
even misleading statistics.
Data warehouse needs consistent integration of
quality data
Data extraction, cleaning, and transformation
comprises the majority of the work of building a
data warehouse
Data Mining: Data Preprocessing lecture notes
Multi-Dimensional Measure of Data
Quality
A well-accepted multidimensional view:
Accuracy
Completeness
Consistency
Timeliness
Believability
Value added
Interpretability
Accessibility
Broad categories:
Intrinsic, contextual, representational, and
accessibility
Data Mining: Data Preprocessing lecture notes
Major Tasks in Data
Preprocessing
Data cleaning
Fill in missing values, smooth noisy data, identify or
remove outliers, and resolve inconsistencies
Data integration
Integration of multiple databases, data cubes, or files
Data transformation
Normalization and aggregation
Data reduction
Obtains reduced representation in volume but produces
the same or similar analytical results
Data discretization
Part of data reduction but with particular importance,
especially for numerical data
Data Mining: Data Preprocessing lecture notes
Forms of Data Preprocessing
Data Mining: Data Preprocessing lecture notes
Data Preprocessing
Why preprocess the data?
Descriptive data summarization
Data cleaning
Data integration and transformation
Data reduction
Summary
Biomedical Data Mining: Data Preprocessing lecture notes
Mining Data Descriptive
Characteristics
Motivation
To better understand the data: central tendency,
variation and spread
Data dispersion characteristics
median, max, min, quartiles, outliers, variance, etc.
Numerical dimensions correspond to sorted intervals
Data dispersion: analyzed with multiple granularities of
precision
Boxplot or quartile analysis on sorted intervals
Dispersion analysis on computed measures
Folding measures into numerical dimensions
Boxplot or quantile analysis on the transformed cube
Data Mining: Data Preprocessing lecture notes
Measuring the Central Tendency
1 n x
x xi
Mean (algebraic measure) (sample vs. population):
n i 1 N
n
Weighted arithmetic mean: w x i i
Trimmed mean: chopping extreme valuesx
i 1
n
w i
Median: A holistic measure i 1
Middle value if odd number of values, or average of the
middle two values otherwise
Estimated by interpolation (for grouped data): n / 2 ( f )l
Mode median L1 ( )c
f median
Value that occurs most frequently in the data
Unimodal, bimodal, trimodal
Empirical formula: mean mode 3 (mean median)
Data Mining: Data Preprocessing lecture notes
Symmetric vs. Skewed
Data
Median, mean and mode of
symmetric(normal), positively
and negatively skewed data
Data Mining: Data Preprocessing lecture notes
Measuring the Dispersion of
Data
Quartiles, outliers and boxplots
Quartiles: Q1 (25th percentile), Q3 (75th percentile)
Inter-quartile range: IQR = Q3 – Q1
Five number summary: min, Q1, M, Q3, max
Boxplot: ends of the box are the quartiles, median is marked,
whiskers, and plot outlier individually
Outlier: usually, a value higher/lower than 1.5 x IQR
Variance and standard deviation (sample: s, population: σ) 2 1 n 2
1 n
( xi ) xi 2
2
Variance: (algebraic, scalable computation) N i 1 N i 1
1 n 1 n 2 1 n 2
2
s
n 1 i 1
2
( xi x ) [ xi ( xi ) ]
n 1 i 1 n i 1
Standard deviation s (or σ) is the square root of variance s2 (or σ2)
Data Mining: Data Preprocessing lecture notes
Boxplot Analysis
Five-number summary of a distribution:
Minimum, Q1, M, Q3, Maximum
Boxplot
Data is represented with a box
The ends of the box are at the first and third
quartiles, i.e., the height of the box is IRQ
The median is marked by a line within the
box
Whiskers: two lines outside the box extend
to Minimum and Maximum
Data Mining: Data Preprocessing lecture notes
Data Preprocessing
Why preprocess the data?
Descriptive data summarization
Data cleaning
Data integration and transformation
Data reduction
Summary
Data Mining: Data Preprocessing lecture notes
Data Cleaning
Importance
“Data cleaning is one of the three biggest
problems in data warehousing”—Ralph Kimball
“Data cleaning is the number one problem in
data warehousing”—DCI survey
Data cleaning tasks
Fill in missing values
Identify outliers and smooth out noisy data
Correct inconsistent data
Resolve redundancy caused by data
integration Data Mining: Data Preprocessing lecture notes
Missing Data
Data is not always available
E.g., many tuples have no recorded value for several
attributes, such as patient’s age in Disease diagnosis
data
Missing data may be due to
equipment malfunction
inconsistent with other recorded data and thus deleted
data not entered due to misunderstanding
certain data may not be considered important at the
time of entry
Missing data may need to be inferred.
Data Mining: Data Preprocessing lecture notes
How to Handle Missing Data?
Ignore the tuple: usually done when class label is missing
(assuming the tasks in classification—not effective when the
percentage of missing values per attribute varies
considerably.
Fill in the missing value manually: tedious + infeasible?
Fill it in automatically with
a global constant : e.g., “unknown”, a new class?!
the attribute mean
the attribute mean for all samples belonging to the same
class: smarter
the most probable value: inference-based such as
Biomedical Data Mining: Data Preprocessing lecture notes
Bayesian formula or decision tree
Noisy Data
Noise: random error or variance in a measured
variable
Incorrect attribute values may due to
faulty data collection instruments
data entry problems
data transmission problems
technology limitation
inconsistency in naming convention
Other data problems which requires data cleaning
duplicate records
incomplete data
inconsistent data
Biomedical Data Mining: Data Preprocessing lecture notes
How to Handle Noisy Data?
Binning
first sort data and partition into (equal-frequency)
bins
then one can smooth by bin means, smooth by
bin median, smooth by bin boundaries, etc.
Regression
smooth by fitting the data into regression
functions
Clustering
detect and remove outliers
Combined computer and human inspection
detect suspicious values and check by human
(e.g., deal with
Datapossible outliers)
Mining: Data Preprocessing lecture notes
Simple Discretization Methods:
Binning
Equal-width (distance) partitioning
Divides the range into N intervals of equal size: uniform grid
if A and B are the lowest and highest values of the attribute,
the width of intervals will be: W = (B –A)/N.
The most straightforward, but outliers may dominate
presentation
Skewed data is not handled well
Equal-depth (frequency) partitioning
Divides the range into N intervals, each containing
approximately same number of samples
Good data scaling
Data Mining: Data Preprocessing lecture notes
Managing categorical attributes can be tricky
Binning Methods for Data Smoothing
Sorted data for drug prices (in dollars): 4, 8, 9, 15, 21, 21,
24, 25, 26, 28, 29, 34
* Partition into equal-frequency (equi-depth) bins:
- Bin 1: 4, 8, 9, 15
- Bin 2: 21, 21, 24, 25
- Bin 3: 26, 28, 29, 34
* Smoothing by bin means:
- Bin 1: 9, 9, 9, 9
- Bin 2: 23, 23, 23, 23
- Bin 3: 29, 29, 29, 29
* Smoothing by bin boundaries:
- Bin 1: 4, 4, 4, 15
- Bin 2: 21, 21, 25, 25
- Bin 3: 26, 26, 26, 34
Data Mining: Data Preprocessing lecture notes
Regression
Y1
Y1’ y=x+1
X1 x
Data Mining: Data Preprocessing lecture notes
Cluster Analysis
Data Mining: Data Preprocessing lecture notes
Data Preprocessing
Why preprocess the data?
Data cleaning
Data integration and transformation
Data reduction
Summary
Data Mining: Data Preprocessing lecture notes
Data Integration
Data integration:
Combines data from multiple sources into a
coherent data store
Schema integration: e.g., [Link]-id [Link]-#
Integrate metadata from different sources
Entity identification problem:
Identify real world entities from multiple data
sources, e.g., Bill Clinton = William Clinton
Detecting and resolving data value conflicts
For the same real world entity, attribute values
from different sources are different
Possible reasons: different representations,
different scales, e.g., Sackets vs. Packets
Data Mining: Data Preprocessing lecture notes
Data Transformation
Smoothing: remove noise from data
Aggregation: summarization, data cube
construction
Generalization: concept hierarchy climbing
Normalization: scaled to fall within a small,
specified range
min-max normalization
z-score normalization
normalization by decimal scaling
Attribute/feature construction
New attributes constructed from the given ones
Data Mining: Data Preprocessing lecture notes
Data Transformation: Normalization
Min-max normalization: to [new_minA, new_maxA]
v minA
v' (new _ maxA new _ minA) new _ minA
maxA minA
Ex. Let income range $12,000 to $98,000 normalized to
73,600 12,000
[0.0, 1.0]. Then $73,000 is mapped (1.0 0) 0 0.716
98,000 to
12,000
Z-score normalization (μ: mean, σ: standard deviation):
v A
v'
A
73,600 54,000
Ex. Let μ = 54,000, σ = 16,000. Then 1.225
16,000
Normalization by decimal scaling
v
v' j Where j is the smallest integer such that Max(|ν’|) < 1
10
Data Mining: Data Preprocessing lecture notes
Data Preprocessing
Why preprocess the data?
Data cleaning
Data integration and transformation
Data reduction
Summary
Data Mining: Data Preprocessing lecture notes
Data Reduction Strategies (More reading
needed)
Why data reduction?
A database/data warehouse may store terabytes of data
Complex data analysis/mining may take a very long time
to run on the complete data set
Data reduction
Obtain a reduced representation of the data set that is
much smaller in volume but yet produce the same (or
almost the same) analytical results
Data reduction strategies
Data cube aggregation:
Dimensionality reduction — e.g., remove unimportant
attributes
Feature/attribute selection : e.g subset and heuristic feature
selection
Numerosity reduction — e.g., fit data into models
Discretization and
Data Mining: Data Preprocessing lecture notes
concept hierarchy generation
Data Preprocessing
Why preprocess the data?
Data cleaning
Data integration and transformation
Data reduction
Summary
Data Mining: Data Preprocessing lecture notes
Summary
Data preparation or preprocessing is a big issue for
both data warehousing and data mining
Discriptive data summarization is needed for
quality data preprocessing
Data preparation includes
Data cleaning and data integration
Data reduction and feature selection
Discretization
A lot a methods have been developed but data
preprocessing still an active area of research
Data Mining: Data Preprocessing lecture notes
References
D. P. Ballou and G. K. Tayi. Enhancing data quality in data warehouse environments. Communications of
ACM, 42:73-78, 1999
T. Dasu and T. Johnson. Exploratory Data Mining and Data Cleaning. John Wiley & Sons, 2003
T. Dasu, T. Johnson, S. Muthukrishnan, V. Shkapenyuk.
Mining Database Structure; Or, How to Build a Data Quality Browser. SIGMOD’02.
H.V. Jagadish et al., Special Issue on Data Reduction Techniques. Bulletin of the Technical Committee on
Data Engineering, 20(4), December 1997
D. Pyle. Data Preparation for Data Mining. Morgan Kaufmann, 1999
E. Rahm and H. H. Do. Data Cleaning: Problems and Current Approaches. IEEE Bulletin of the Technical
Committee on Data Engineering. Vol.23, No.4
V. Raman and J. Hellerstein. Potters Wheel: An Interactive Framework for Data Cleaning and
Transformation, VLDB’2001
T. Redman. Data Quality: Management and Technology. Bantam Books, 1992
Y. Wand and R. Wang. Anchoring data quality dimensions ontological foundations. Communications of
ACM, 39:86-95, 1996
R. Wang, V. Storey, and C. Firth. A framework for analysis of data quality research. IEEE Trans. Knowledge
and Data Engineering, 7:623-640, 1995
Data Mining: Data Preprocessing lecture notes