LECTURE 7-8: Data

Preprocessing
 Why pre- Missing Entity Strategies Strategies
process values Identification
Wavelet Normalization
Overview

data? Problem
Data Cleaning

Data Reduction
Data Integration
Noisy data transforms

Data Transformation
Major tasks Redundancy Binning
and PCA
Histogram
correlation Attribute analysis
analysis subset
selection Cluster/Decision
Tuple trees/Correlation
duplication Regression analyses
Data value Histograms Concept
conflict hierarchy
detection Clustering
Sampling

2
 Why Missing Entity Strategies Strategies
preprocess values Identification
Wavelet Normalization
Overview

data? Problem
Data Cleaning

Data Reduction
Data Integration
Noisy data transforms

Data Transformation
Major tasks Redundancy Binning
and PCA
Histogram
correlation Attribute analysis
analysis subset
selection Cluster/Decision
Tuple trees/Correlation
duplication Regression analyses
Data value Histograms Concept
conflict hierarchy
detection Clustering
Sampling

3
Databases are highly susceptible to noisy, missing,
and inconsistent data
Low-quality data will lead to low-quality mining
results

4
To satisfy the requirements of the intended use
Factors of data quality:
Accuracy → lack of due to faulty instruments, errors caused by
human/computer/transmission, deliberate errors …
Completeness → lack of due to different design phases,
optional attributes
Consistency → lack of due to semantics, data types, field
formats …
Timeliness
Believability →how much the data are trusted by users
Interpretability → how easy the data are understood

5 2/26/2013
ThatImprove Quality ofData
Data cleaning → filling in missing values, smoothing
noisy data, identifying or removing outliers, and resolving
inconsistencies
Data integration → include data from multiple sources in
your analysis, map semantic concepts, infer attributes …
Data reduction → obtain a reduced representation of
the data set that is much smaller in volume, while
producing almost the same analytical results
Discretization → raw data values for attributes are
replaced by ranges or higher conceptual levels
Data transformation → normalization

6
 Why Missing Entity Strategies Strategies
preprocess values Identification
Wavelet Normalization
Overview

data? Problem
Data Cleaning

Data Reduction
Data Integration
Noisy data transforms

Data Transformation
Major tasks Redundancy Binning
and PCA
Histogram
correlation Attribute analysis
analysis subset
selection Cluster/Decision
Tuple trees/Correlation
duplication Regression analyses
Data value Histograms Concept
conflict hierarchy
detection Clustering
Sampling

7
Data in the Real World Is Dirty!
incomplete: lacking attribute values, lacking certain
attributes of interest, or containing only aggregate data
e.g., Occupation=“ ” (missing data)
noisy: containing noise, errors, or outliers
e.g., Salary=“−10” (an error)
inconsistent: containing discrepancies in codes or names,
e.g.,
Age=“42”, Birthday=“03/07/2010”
Wasrating “1, 2, 3”, now rating “A,B, C”
discrepancy between duplicate records
Intentional → Jan. 1 aseveryone’s birthday?

8
…fill in missing values, smooth
out noise while identifying
outliers, and correct
inconsistencies in the data

A missing value may not imply

an error in the data!
e.g. driver’s license number

9
Missing Values

Ignore the tuple → not very effective, unless the tuple

contains several attributes with missing values
Fill in the missing value manually → time consuming,
not feasible for large data sets
Use a global constant → replace all missing attribute
values by same value (e.g. unknown)
may mistakenly think that “unknown” isan
interesting concept

10
Missing Values

Use mean or median → For normal (symmetric) data

distributions, the mean is used, while skewed data
distribution should employ the median
Use mean or median for all samples belonging to the
same class as the given tuple → e.g. mean or median
of customers in a certain age group
Use the most probable value → using regression,
inference-based tools such as decision tree
Most popular

11
NoisyData

Noise is a random error or variance in a measured

variable
Data smoothing techniques:
1. Binning
2. Regression
3. Outlier Analysis

12
 NoisyData
1. Binning → smooth a sorted data value by consulting its
“neighborhood”
sorted values are partitioned into a # of “buckets,” or bins →
local smoothing
equal-frequency bins → each bin has same # of values
equal-width bins → interval range of values per bin is constant
Smoothing by bin means → each bin value is replaced by the bin
mean
Smoothing by bin medians → each bin value is replaced by the bin
median
Smoothing by bin boundaries → each bin value is replaced by the
closest boundary value (min & max in a bin are bin boundaries)

13
NoisyData
Example: Sorted data for price (in dollars): 4, 8, 15, 21, 21, 24, 25,
28, 34
Partition into (equal-frequency) bins
Bin 1: 4, 8, 15
Bin 2: 21, 21, 24
Bin 3: 25, 28, 34
Smoothing by bin means
Bin 1: 9, 9, 9
Bin 2: 22, 22, 22
Bin 3: 29, 29, 29
Smoothing by bin boundaries
Bin 1: 4, 4, 15
Bin 2: 21, 21, 24
Bin 3: 25, 25, 34

14
NoisyData

2. Regression → Conform data values to a function

Linear regression → find “best” line to fit two
attributes so that one attribute can be used to
predict the other
More on this later in the lecture
3. Outlier Analysis
Pot er’s Wheel → Automated interactive data
cleaning tool

15
 Why Missing Entity Strategies Strategies
preprocess values Identification
Wavelet Normalization
Overview

data? Problem
Data Cleaning

Data Reduction
Data Integration
Noisy data transforms

Data Transformation
Major tasks Redundancy Binning
and PCA
Histogram
correlation Attribute analysis
analysis subset
selection Cluster/Decision
Tuple trees/Correlation
duplication Regression analyses
Data value Histograms Concept
conflict hierarchy
detection Clustering
Sampling

16
Merging data from multiple data stores
Helps reduce and avoid redundancies and
inconsistencies in the resulting data set
Challenges:
Semantic heterogeneity → entity identification
problem
Structure of data → functional dependencies and
referential constraints
Redundancy

17
Entity IdentificationProblem

Schema integration and object matching

Metadata → name, meaning, data type, and
range of values permitted, null rules for
handling blank, zero, or null values
can help avoid errors in schema integration and
data transformation

18
Redundancy and CorrelationAnalysis

19
Redundancy and CorrelationAnalysis

Gender
male female Total
Fiction 250 200 450

Preferred Non- 50 1000 1050

reading fiction
Total 300 1200 1500

20
Redundancy and CorrelationAnalysis

gender
male female Total
Fiction 250 (90) 200 (360) 450

Preferred Non- 50(210) 1000 (840) 1050

reading fiction
Total 300 1200 1500

21
Redundancy and CorrelationAnalysis

gender
male female Total
Fiction 250 (90) 200 (360) 450

Preferred Non- 50 (210) 1000 (840) 1050

reading fiction
Total 300 1200 1500

22
 correlation matrix
A correlation matrix is simply a table that displays the correlation coefficients for different
variables. The matrix depicts the correlation between all the possible pairs of values in a
table. It is a powerful tool to summarize a large dataset and to identify and visualize patterns
in the given data.
Redundancy and CorrelationAnalysis

Time point AllElectronics HighTech

T1 6 20
T2 5 10
T3 4 14
T4 3 5
T5 2 5

26
MoreIssues

Tuple duplication
e.g. purchaser name and her purchases
Data value conflict
e.g. grading system in two different institutes → A,
B, …versus 90%, 80% …

27
 Why pre- Missing Entity Strategies Strategies
process values Identification
Wavelet Normalization
Overview

data? Problem
Data Cleaning

Data Reduction
Data Integration
Noisy data transforms

Data Transformation
Major tasks Redundancy Binning
and PCA
Histogram
correlation Attribute analysis
analysis subset
selection Cluster/Decision
Tuple trees/Correlation
duplication Regression analyses
Data value Histograms Concept
conflict hierarchy
detection Clustering
Sampling

28
Strategies
Dimensionality reduction → reduce number of attributes
Wavelet transforms, PCA, Attribute subset selection
Numerosity reduction → replace original data volume by smaller
data representation
Parametric → a model is used to estimate the data - only
the data parameters are stored
Regression
Nonparametric → store reduced representations of the
data
Histograms, clustering, sampling
Compression → transformations applied to obtain a
“compressed” representation of original data
Lossless, Lossy

29
Parametric and non-parametric tests
If you want to calculate a hypothesis test, you must first check the prerequisites of the
hypothesis test. A very common requirement is that the data used must be subject to some
distribution, usually the normal distribution. If your data are normally distributed,
parametric tests can usually be used, if they are not normally distributed, non-parametric
tests are usually used.

Parametric tests

If the data are normally distributed, parametric tests such as the t-test, ANOVA or Pearson
correlation are used.

Non-parametric tests

If the data are not normally distributed, the nonparametric tests are used. These are for
example the Mann-Whitney U-Test or the Wilcoxon-Test.

Nonparametric tests are therefore used when the scale level is not metric, the true
distribution of the random variables is not known, or the sample is simply too small to
assume a normal distribution. To assume a normal distribution, the sample should at least
be larger than 30 cases.

Thus, non-parametric tests are more robust than parametric tests and can be calculated in
significantly more situations. Parametric tests, however, have a greater statistical power
than the non-parametric tests. Therefore, if the assumptions for a parametric test are met,
it should always be used.

The following table lists the most common parametric and nonparametric tests. Depending
on the number of samples and whether they are dependent or independent, there is a
parametric and a nonparametric test.
Regression

31
Regression
Data is modeled to fit a straight line
A random variable y (response variable), can be
modeled as a linear function of another random
variable x (predictor variable)
Regression line equation → y = wx + b
w and b are regression coefficients → they specify the
slope of the line and y-intercept
Solved for by the method of least squares →minimize
error between actual line separating data and estimate
of the line (best-fitting line)

32
Regression
Example:
X Y
1.00 1.00
2.00 2.00
3.00 1.30
4.00 3.75
5.00 2.25

33
Regression

X Y
1.00 1.00
2.00 2.00
3.00 1.30
4.00 3.75
5.00 2.25

34
 Why Missing Entity Strategies Strategies
preprocess values Identification
Wavelet Normalization
Overview

data? Problem
Data Cleaning

Data Reduction
Data Integration
Noisy data transforms

Data Transformation
Major tasks Redundancy Binning
and PCA
Histogram
correlation Attribute analysis
analysis subset
selection Cluster/Decision
Tuple trees/Correlation
duplication Regression analyses
Data value Histograms Concept
conflict hierarchy
detection Clustering
Sampling

37
Transformation Strategies
Smoothing → binning, regression
Attribute construction
Aggregation
Normalization → raw values of a numeric attribute
(e.g. age) replaced by interval labels (e.g. 0–10, 11–20)
or conceptual labels (e.g., youth, adult, senior)
Concept hierarchy → e.g. street generalized to higher-
level concepts (city or country)

38
Transformation byNormalization
To help avoid dependence on the choice of
measurement units
Give all attributes equal weight
Methods:
min-max normalization
z-score normalization

39
Cleaning Integration Reduction Transformation/Discretization
Binning Binning
Regression Regression Regression
Correlation analysis Correlation
Histograms Histogram analysis
Clustering Clustering
Attribute construction Attribute construction

Aggregation
Normalization
Outlier analysis
Wavelet transforms
PCA
Attribute subset selection

Sampling
Concept hierarchy