4.
2 Data Preprocessing Methods
vulnerable to missing values, noise and inconsisterncy and the quality of data
Raw data is highly
the quality of data, in order to improve
affeets the data mining results. So,there is a necd to improve
pre-processed so as to enhance its quality
mining results. For achieving better results, raw data is
and make it error free. This cases the mining process.
preprocessing is performed.
As depicted in Figure 4.1 the various stages in which data
" Data Cleaning
" Data Integration
" Data Transformation
" Data Reduction
(Water to clean dirty-looking data) Clean' Looking data)
Data cleaning
(Showsoap suds on data)
Data Integration
Data Transformation -2, 32, 100, 59, 48 -0.02. 0.32, 1.00, 0.59, 0.48
Data Reduction A1 A2 A3 ... A126 A1 A3 A115
T1 T1
T2 T4
T3
T4
T1456|
T2000|
Figure 4.1 Various stages of preprocessing
Let'sdiscuss the details of each phase one byone.
Data Preprocessing 59
4.2.1 Datacleaning
Cirst the raw data or noisy data goes
olues are filled, noisy data is through theprocess of cleansing. In data
and removed in order to clean smoothened, inconsistencies cleansing missing
are resolved, and outliers
the data. To claborate further: are identified
Handling missing values
It is often found that many of the database tuples or
some attributes. Such cases of records do not
missing values are filled by different have anyasrecorded values for
i. Fillin the missing value methods, described below.
and time consuming andmanually: Naturally, manually filling each missing value is
so it is practical only
There are other methods to deal with when the missing values are few in laborious
large or when the missing values are the problem of missing values when the dataset is very
number.
ii. Use ofsome global very many.
constant in place of missing value: In this
by some global label such as
Unkown' or -oo,
method, missing values are replaced
with the missing values, it should be Although one of the easiest approaches to deal
avoided
repetitive occurrences of global labels such as when mining program presents a pattern due to
with caution. Unknown'. Hence, this method should be used
iii. Use the attribute mean to fill in
the missing value: Fill in the
with the mean of other data values of
the
missing values for each attribute
values in a dataset. same attribute. This is a better way to handle missing
iv. Use some other value which is
high in probability to fill in the missing
with values determined by tools value:
is to fill in the missing values Another efficient method
Decision Tree Induction or other inference such as Bayesian Formalism or
uses most of the information based tools. This is one of the best
already present to predict the missing methods, as it
biased like previous methods. Theonly values, although it is not
the analysis. difficulty with this method complexity in performing
is
V.
Ignore the tuple: If the tuple contains more
not applicable, then the best than one missing value and all other
strategy to cope with missing values is to methods are
This is commonly used if the class ignore the whole tuple.
most of the attributes. This label goes missing or the tuple contains
method should not be used, if the missing values for
missing per attribute varies percentage of values that are
So, we can significantly.
conclude that use the attribute mean to fill in the
ennique used by most data mining tools to
missing value is most common
use handle the missing values. However, one can always
knowledge of fprobability to fill these values.
Handling
Most
noisy data
data mining8 algorithms are affected adversely due to noisy data.
unwanted variance or Some random error that occurred in a
from the data by the method of 'smoothing. The measurable
The noise can be detined as
variable. Noise is removed
methods used for data snoothing are as tollows:
i. Binning methods
divide thc values ofan attribute into bins or buckets. It is commonlv
The Binning method is used to type. For example, it may be nccessary to convert a
tvpc of attribute to another cool, warm
used toconvert one nominal attribute with values cold,
attribute like temperature to a
real valuednumeric
also called 'discretizing' a numeric attribute. There are two
and hot betore its processing. This is
equalinterval and equal frequency.
tpes of discretization,namely, calculate the samples into the appropriate bin
we a bin size and then put
Incqual interval binning, vary, with our goal being to choose bin sizes so
allow the bin sizes to
Inequal frequency binning, we number of samples in it. The idea is that if each bin has the sarne
that every bin has about the same
samples, no bin, or sample, will have greater or lesser impact on the results of data mining.
number of 50 students. . The process divides
marks of
To understand this process, consider a dataset of the
example, 10 bins.
this dataset on the basis of their marks into, for this
10-20, 20-30, 30-40, 40-50, 50
In case ofequal interval binning, we will create bins from 0-10,
some
60, 60-70, 70-80, 80-90, 90-100. If most students commonly have marks between 60 to 80,
bins may be full and most bins may have very few entries e.g., 0-10, 10-20, 90-100.
Thus, it might be better to divide this dataset on the equal frequency basis. It means that with
the same 50 students in class and we want to put these into 10 binson the basis of their marks then
instead of creating the bins for marks like 0-10, 10-20 and so on, here wewill first sort the records
of students on the basis of their marks in descending order (or ascending order as we prefer). The
first 5 students having highest marks will put into one bin and next 5 students on the basis of their
marks will put into another and so on. If our boundary students have same marks then bin range
can be shifted to accommodate students with the same marks into one common bin. For example,
let us suppose that after arranging the data in descending order of marks and we found that marks
of 5th and 6th students are same of 85. Then we cannot put one
student in one bin and other in a
different bin because both have the same marks. So, we either shift our bin range may be
in this case) to accommodate the first 4 students in one bin up (i.e., 86
and next 5 into another. Similarly, we
can shift our bin range dovwn to accommodate first 6
students in one bin (i.e., 85 in this case so that
5th and 6th student falls in same bin) and next 5
into another.
most of bins will have a count of approximately 5, while in Thus, in this case of equal frequency
case of equal interval some bins will be
heavily loaded while most will be lightly loaded.
Thus, the idea of having same number of samples in each
bin
will have greater or lesser impact on the results of data mining. works better as no bin, or sample,
iü. Clustering or outlier analysis
Clustering or outlier analysis is amethod that allows detection of outliers by
values which are common or similar are organized clustering. In clustering,
lie intogroups
outside these clusters are termed as outliers or noise.
or 'clusters', and those values which
iii. Regression
Regression is another such method which allowsdata smoothing by fitting it to
example, Linear Regression is one of the most used methods that aims at some function. For
finding the most suitable
eprocessing 61
line to fit values ofEtwo variables or attributes
Lavalue of other variable using the first one.(ie.,best fit). The primary purpose of this is to predict
Similarly, Multiple Regression is used when nore
Asn fyo variables are involved.
Regression
lata andhence smoothens the datasect using allows data fitting which in turn rernoves noise frorn
mathematical equations.
[Link] computer and human inspection
Ilsingboth computers and human inspection one can
dctect suspicious values and outliers.
Handling of inconsistent data
Many times data inconsistencies are encountered when data is recorded during some
Such inconsistencies can be manually removed by using transaction.
that have been made at the time of data external references. As an example: errors
entry be corrected manually by performing a paper
operation. trace
4.2.2 Data integration
A most necessary step to be taken during data
analysis is Data Integration. Data integration is a
process which combines data from a plethora of sources (such as
data cubes) into a unified data store. multiple databases, flat files or
The example of the University Database system discussed under
of data integration. section 4.1 refers to the issues
During data integration, a number of tricky issues have to be considered. For example, how
the data analyst or the analyzing machine be sure that student_id of one does
of another database refer to the same entity? This is referred to as the
database and student_number
problem of
Solution to the problem lies with the term 'metadata'. Databases and data entity identification.
metadata, which is data about data. This metadata is taken as a reference and warehouses consist of
referred by the data
analyst to avoid errors during the process of data integration.
Another such issue which may be caused due to schema integration is redundancy. In the
ofdatabase, an attribute is said to be redundant if it is language
derivable from some other table (of the same
database). Mistakes in attribute naming can also lead to data redundancies in the resulting
We use a number of tools to performn data integration from different dataset.
sources into one unified schema.
4.2.3 Data transformation
When the value of one attribute is small as compared to other attributes, then that
not have attribute will
than othermuch influence on mining of information, since the values of this attribute were sinaller
attributes andthe variation within the attribute will also be small.
Thus, data transformation is a process in which data is consolidated or transtormedinto some
Other standard forms which are better suited for data
nining.
example,the dataset given in Figure 4.2 is for the chemicalcomposition of wine samples.
Note that the values for different attributes cover arange of six orders of magnitude. It turns out
that data mining algorithms struggle with numerie attributes that exhitbit such ranges of values.
Sepuo
65 0.99340
015 2.1 0.054
106 .99320 3.
2.3 0.058 17 Lo991543.70.24140
0.17
42
040 0.(00 2.1
1.6
0.048
0.048
16
26 60 [Link] file available in
0.01
04000 050 4.3 0.046 29 80 [Link] at
0.420 0.24 2.0 0.060 19 [Link]/Datasets
0.740 0.00 1.2 0.041 16 46
0.99380 3.75 O.48 10.5
1.020 0.04 1.4 0.045 41 85
0.99340 3.74 O.62 11.5
1.040 0.24 1.6 0.050 32 96
13.6 R
0044 18 88 O.99157 3.68 0.73
5.1 O420 0.00 1.8 R 6
0.992 10 3.90 O.62 12.8
51 0.470 O.02 1.3 0.034 18
0.87 12.9 R
5.1 0.510 0.18 2.1 0.042 1 101 0.99240 3.46
Max Min = 5%
10 range of of average
magnitudes
Figure 4.2 Chemical compositionof wine samples
All attributes should be transformed to asimilar scale for clustering to beeffective unless we wish
to give more weight to some attributes that are comparatively large in scale. Commonly,we use two
techniques toconvert the attributes: Normalization and Standardization are the most popular and
widely used data transformation methods.
Normalization
Incase of normalization, all the attributes are converted to a normalized score or to a range (0, 1).
The problem of normalization is an outlier. If there is an outlier, it willtend to crunch all of the
other values down toward the value of zero. In order to understand this, let's suppose the range of
students' marks is 35 to 45 out of 100. Then 35 willbe considered as Oand 45 as 1, and students
will be distributed between 0 to 1 depending upon their marks. But if there is one student having
marks 90, then it willact as an outlier and in this case, 35 will be considered as Oand 90 as 1. Now,
it willcrunch most of the values down toward the value of zero. In this scenario, the solution is
standardization.
Standardization
In case of standardization, the values are all spread out so that we have a standard deviation of 1.
Generally, there is no rule for when to use normalization versus standardization. However, if vour
data has outliers, use standardization, otherwise use [Link] standardization tends to
make theremaining values for allof the other attributes fall into similar ranges since all attributes
will have the same standard deviation of 1.
4.2.4 Data reduction
It is often seen that when the complex data analysis and mining processes are carried out over
humongous datasets, they take such a long time that the whole data mining or analysis process
becomes unviable. Data recduction techniqucs come to the rescue in such situations. Using data
reduction techniquesa datasett can be represented in areduced manner without actually cornprornising
theintegrity of originaldata.
Data reduetion isall about reducing the dimensions (referring to the total number of attributes)
reducing the voume. Moreover, mining when carried out on reduced datasets often results in
hetter aurary and proves to be more efficient. There are many methods to reduce large datasets
to icld usetul knowledge. Afew among them arc:
iDimension reduction
In data warehousing, dimension' equips us with structured labeling
information. But not all
Principal
dimensions(attributes) are necessary at a time. Dimension reduction uses algorithm such asdetect and
Component Analysis (PCA) and others. With the usage of such algorithms one
can
remove redundant and weakly relevant, attributes or dimensions.
iü. Numerosity reduction
data representation for reducing the
It is a technique which is used to choose smaller forms of
dataset volume.
ii. Data compression
also use data comnpression techniques to reduce the dataset size. These techniques are
We can
where some encoding mechanisms (e.g.
classified as lossy and loseless compression techniques
Huffman coding) are used.